The College of Engineering provides access to a Great Lakes resource account that can be used by any Engineering student, faculty, or staff member. This shared resource account is not intended to replace or supplement other departmental or group accounts. Research groups who need more computation than what is provided under the usage limits (listed below), or who need their jobs to start earlier than what they would use under this shared account, should obtain their own Great Lakes account.
General U-M Great Lakes accounts can only be obtained by research groups across University departments, centers, institutes, or other units. In addition, graduate students can use Rackham Graduate Student Research Grants to purchase their own, private Great Lakes accounts.
Overall account limit:
90 cores, 450 GB RAM, 2 GPUs
Individual usage limits (maximums per person):
36 cores, 180 GB RAM, 2 GPUs
Job cost equivalent of 5 core*months remaining across all running jobs
GPU runtime of 48 gpu*hours remaining across all running jobs
Approved uses of the engin1 account on Great Lakes include, but are not limited to:
- Testing the service to determine whether to purchase an account.
- Experimentation and exploration on an ad hoc, small-scale basis of questions not necessarily tied to any particular research project; without needing to obtain funding for a paid resource account.
Note that Principal Investigators (PIs) who have not used Great Lakes can also request a one-time trial account by contacting firstname.lastname@example.org. The trial gives $150 worth of compute, and cannot be used after one (1) month.
The College of Engineering has set usage limits on its shared Great Lakes resource account, engin1, in order to prevent a single individual or small group from monopolizing the account for extended periods of time to the detriment of others:
An individual may use up to 36 cores or up to 180 GB of memory across all of their running jobs at any point in time on engin1. Additionally, individuals are limited to having no more than the cost of 5*core months (3,600*core hours) worth of jobs running at any one time. The limits are calculated by summing the product of the remaining walltime and the cost of the resources (CPU cores, RAM, and GPU) used. Assuming standard, non-GPU, jobs with 5 GB/core, 5 core*months are sufficient to run a 10 core job for 14 days, or a 20 core job for 7.5 days, or a 36 core job for 4 days, or two 18 core jobs for 4 days, or many other combinations.
Two GPUs are available in engin1 on the gpu partition. Individuals are limited to having no more than 48 gpu*hours worth of jobs running at any one time, which means the maximum walltime is 24 hours when using two GPUs, and when one GPU is used, it is 48 hours. Both the GPUs in engin1 may be used at the same time by an individual, either in the same job or in two separate jobs. GPU jobs do count towards the limit of the cost of 5*core months. Please be aware that each GPU, plus associated RAM and cores could count as much as 11.5 times a regular job running on a single CPU core plus associated RAM.
Ideally, individuals should use only the resources they need. Individuals should not use more than the cost of 5*core months, which is about $54 in a given month, but this is not set up as a limit to allow individuals some flexibility. If an individual user goes over this limit more than once, they will receive an email notification, and further user level limits may be imposed on that individual. Additionally, if the individual is a student and does not respond to the notification, and continues to exceed the limit, the individual’s PI or advisor will be contacted.
Q & A
- Why was the individual usage limit changed from being solely based on CPU consumption?
The limit was changed from 5*core months across all running jobs to the equivalent cost of 5*core months across all running jobs, to encourage all users to make the most economical use of the limited resources available. While individual users do not directly pay for engin1, the College of Engineering does pay for all usage on the account from a limited budget, and jobs with equivalent compute can cost more or less depending on how they are submitted. A job using a single core with 180GB costs more on the standard partition than on the largemem partition, but the compute is exactly the same. For most users, the limit based on cost is effectively the same as the original CPU-time limit.
- How do you calculate the cost of a job?
It is difficult to calculate the exact cost of a job because of runtime, but you can estimate the cost using the “my_job_estimate” script:
[user@gl-login1 ~]$ my_job_estimate --cores 1 --memory 180g --time 24:00:00 \ --partition standard | grep "\\$" Total: $15.93 for 24.0 hours. [user@gl-login1 ~]$ my_job_estimate --cores 1 --memory 180g --time 24:00:00 \ --partition largemem | grep "\\$" Total: $8.52 for 24.0 hours.
- Why do jobs take a long time to start on the engin1 account?
The account has resources limits to control the cost and is shared between many users. Individual jobs may have to wait in the queue when the account is busy and other jobs are also waiting to run.
- May I use engin1 to do a homework assignment?
Since this is a shared account, it is not suitable for course work. Your job may wait in the queue for a long time and you may miss the due date of your assignment. Class accounts should be used for course work. If your class does not have an account, please ask your instructor to contact: email@example.com
- May I use engin1 to do research?
Research computing jobs related to an established research project should be run using the paid Great Lakes account set up by the PI of that research project. You may use engin1 to try out Great Lakes on a small scale and see if it is suitable for use with a new research project before your PI sets up their paid resource account.
Send any questions about the College of Engineering’s shared Great Lakes account to: firstname.lastname@example.org