test: add testcase for disable some model providers

autoqa/tested/models/disable-selected-providers.txt

@@ -0,0 +1,78 @@
+prompt = """
+You are a GUI agent. You are given a task and your action history, with screenshots. You need to perform the next action to complete the task. 
+
+## Output Format
+```
+Thought: ...
+Action: ...
+```
+
+## Action Space
+
+click(start_box='<|box_start|>(x1,y1)<|box_end|>')
+left_double(start_box='<|box_start|>(x1,y1)<|box_end|>')
+right_single(start_box='<|box_start|>(x1,y1)<|box_end|>')
+drag(start_box='<|box_start|>(x1,y1)<|box_end|>', end_box='<|box_start|>(x3,y3)<|box_end|>')
+hotkey(key='')
+type(content='') #If you want to submit your input, use "\
+" at the end of `content`.
+scroll(start_box='<|box_start|>(x1,y1)<|box_end|>', direction='down or up or right or left')
+wait() #Sleep for 5s and take a screenshot to check for any changes.
+finished()
+call_user() # Submit the task and call the user when the task is unsolvable, or when you need the user's help.
+
+
+## Note
+- Use Chinese in `Thought` part.
+- Summarize your next action (with its target element) in one sentence in `Thought` part.
+
+## User Instruction
+
+You are disabling specific Model Providers in the Jan application to verify they are hidden from the sidebar list.
+
+TARGET PROVIDERS TO DISABLE: Llama.cpp, OpenAI, Anthropic
+
+Step-by-step instructions:
+
+1. Open the Jan application.
+   - If a dialog appears in the bottom-right corner titled **"Help Us Improve Jan"**, click **Deny** to dismiss it before continuing.
+   - If **New Version Available** popup appears on app launch (older versions), click **Remind Me Later** to dismiss it before continuing.
+
+2. Navigate to Model Providers settings:
+   - In the bottom-left menu, click **Settings**.
+   - In the left sidebar, click **Model Providers**.
+
+3. Disable selected providers:
+   - For each of the following providers on the right panel (Llama.cpp, OpenAI, Anthropic):
+     - Locate the provider row.
+     - On the right side of the row, find the enable/disable chip/toggle (enabled by default).
+     - If it is enabled, click the chip/toggle to disable it.
+     - Immediately call `wait()` once to allow the UI to update and the next screenshot to be captured for the model.
+     - If it is already disabled, leave it as is (no click). Still call `wait()` once to confirm state in the next screenshot.
+
+4. Verify providers are hidden in the sidebar list:
+   - Call `wait()` once more to ensure the sidebar reflects the latest state.
+   - Check the left sidebar list under **Model Providers**.
+   - Confirm that the entries for **Llama.cpp**, **OpenAI**, and **Anthropic** are not visible in the sidebar after disabling.
+
+5. Return result:
+   - If all three providers (Llama.cpp, OpenAI, Anthropic) are disabled and not visible in the sidebar list, return:
+     {"result": True}
+   - If any of the three providers is still visible in the sidebar list after being disabled, return:
+     {"result": False}
+
+CRITICAL INSTRUCTIONS FOR FINAL RESPONSE:
+- You MUST respond in English only, not any other language
+- You MUST return ONLY the JSON format below, nothing else
+- Do NOT add any explanations, thoughts, or additional text
+
+   - On success, return:
+     {"result": True}
+   - On failure, return:
+     {"result": False}
+
+IMPORTANT: 
+- Your response must be ONLY the JSON above
+- Do NOT add any other text before or after the JSON
+"""
+

autoqa/windows-qa-checklist.md

@@ -15,7 +15,7 @@ Before testing, set-up the following in the old version to make sure that we can
 - [x] Create a new chat with the custom assistant 🔥🔥🔥
 - [x] Change the `App Data` to some other folder
 - [x] Create a Custom Provider 🔥🔥 (Not verified yet)
-- [x] Disabled some model providers 🔥🔥🔥
+- [x] Disabled some model providers ✅ (Intermittently work)
 #### Validate that the update does not corrupt existing user data or settings (before and after update show the same information):
 - [x] Threads
 	- [x] Previously used model and assistants is shown correctly