High Performance Computing (HPC) environments are essential for research, engineering, and data-intensive workloads. To efficiently manage compute resources and job submissions, organizations rely on robust scheduling and orchestration tools. In this blog post, we’ll explore how to use Copilot Agent in Visual Studio Code (VSCode) to build an Open OnDemand application that submits Slurm jobs in a CycleCloud Workspace for Slurm (CCWS) environment. We’ll start with a brief overview of CCWS and Open OnDemand, then dive into the integration workflow.

What is CycleCloud Workspace for Slurm (CCWS)?

CycleCloud Workspace for Slurm (CCWS) is a cloud-based solution provided by Microsoft Azure that simplifies the deployment, management, and scaling of Slurm clusters in the cloud. Slurm is a widely-used, open-source workload manager designed for large-scale compute clusters. With CCWS, users can take advantage of Azure’s elasticity and scalability, automating the provisioning of compute resources according to workload demands. This enables research teams and organizations to run HPC jobs efficiently without the operational overhead of managing on-premises clusters.

What is Open OnDemand?

Open OnDemand is an open-source web portal that provides remote access to HPC resources. It offers users a modern, user-friendly interface to submit jobs, access files, and use interactive applications such as VSCode and remote desktops. By abstracting away complex command-line interactions, Open OnDemand empowers a broader range of users to leverage HPC systems effectively.

Leveraging Copilot Agent in VSCode

GitHub Copilot Agent in VSCode is an AI-powered assistant that helps developers write and manage code more efficiently. With Copilot, you can generate code snippets, automate repetitive tasks, and get intelligent code suggestions—all within the familiar VSCode environment.

Building an Open OnDemand Application for CCWS

To create an Open OnDemand app that submits Slurm jobs in a CCWS environment, you can use Copilot Agent in VSCode to streamline the development process. Here’s how you can approach this workflow which is also presented in the attached video:

1. Enabling the application development environment:

To set up the application development environment, access the Open OnDemand VM and run the following commands:

sudo mkdir -p /var/www/ood/apps/dev/

cd /var/www/ood/apps/dev/

sudo ln -s /shared/home//ondemand/dev gateway

2. Refresh the OOD home page associated with to display the Develop menu. Select “My Sandbox Apps” to clone and create new applications.

3. Start Visual Studio Code on the Login Node and set up the Github Copilot extension.

1. In VSCode, create a new directory for your OnDemand app (e.g., my_slurm_app).


2. Open a Copilot chat and enter these lines :

Create an Open OnDemand application to submit a Slurm batch job on the htc, hpc, or gpu partitions. The app should allow selection of only the number of nodes and maximum walltime. Ensure the job is exclusive.

1. When asked to run a local command, click continue.


2. After a period of processing, files will be generated and a summary will be presented.


3. Click the “Keep” button next to the chat to save all created files.

1. The newly developed application should now be available within your Open OnDemand sandbox environment.

1. Use the Copilot Agent to resolve problems.


2. Use the new application to submit a test job

Conclusion

By combining the power of Copilot Agent in VSCode with Open OnDemand and CycleCloud Workspace for Slurm, you can build user-friendly HPC applications that streamline job submission and resource management. This approach reduces the complexity of developing and deploying HPC workflows, enabling researchers and engineers to focus on their core work while leveraging the scalability of the cloud.