Last December Amazon announced its new EBS gp3 volumes, which offer better performance and a cost saving of 20% compared to those that have been used until now (gp2). Well, after successfully testing these new volumes with multiple clients, I can do nothing but recommend their use, because they are all advantages and in these 2 and a half months that have passed since the announcement I have not noticed any problems or side effects.
Making the change is extremely simple, just select a volume from the AWS console and change its type like this:
However, doing it this way when you have lots of clients that may have hundreds or thousands of these volumes, makes this a tedious, time-consuming and error-prone task, as we may inadvertently select a volume that was not gp2, or change it to another type that is not the expected gp3. And once we have made a mistake, the volume remains in the “modifying” state for quite some time and then in the “optimizing” state for even longer, which will require us to move on to something else and then remember to review all the volumes and correct those where we made a mistake.
To avoid these problems when we have a large number of EBS volumes, I wrote a very simple bash script that completely automates the task, saving us a lot of time and tedium and protecting us from making mistakes:
#! /bin/bash region='us-east-1' # Find all gp2 volumes within the given region volume_ids=$(/usr/bin/aws ec2 describe-volumes --region "${region}" --filters Name=volume-type,Values=gp2 | jq -r '.Volumes[].VolumeId') # Iterate all gp2 volumes and change its type to gp3 for volume_id in ${volume_ids};do result=$(/usr/bin/aws ec2 modify-volume --region "${region}" --volume-type=gp3 --volume-id "${volume_id}" | jq '.VolumeModification.ModificationState' | sed 's/"//g') if [ $? -eq 0 ] && [ "${result}" == "modifying" ];then echo "OK: volume ${volume_id} changed to state 'modifying'" else echo "ERROR: couldn't change volume ${volume_id} type to gp3!" fi done
The only requirement is to have the jq tool to better handle the JSON returned by the aws command. It can be installed by simply running apt install jq or yum install jq, as it is included in the repositories of all Linux distributions.
Finally, it should be noted that this script will cause the default values for the IOPS (3000) and Throughput (125 MB/s) parameters to be set, but as these far exceed the performance offered by the gp2 volumes, this should not be a problem.
Needless to say, this same script can be used to massively change EBS volumes of other types, such as io1, io2, sc1, st1, etc.
Hope it helps you save time and headaches!
18 comments
Join the conversationZachary Armstrong - 27/03/2021
You should remove the ! from the error output message, as it will crap out in bash.
Daniel - 27/03/2021
The ! is not interpreted by bash as it’s part of a string between quotes. I don’t see any problem by using it, but feel free to remove it if it’s causing any problem to you!
Zachary Armstrong - 31/03/2021
That’s not true. If it’s enclosed in single quotes, you can use the !, but if you use double quotes, as you did in the scrip, you have to slash escape it \!. If it’s working for you with double quotes, that means you have history expansion disabled, as ! is the default history expansion character. History expansion is on by default, so you (or someone else) must have purposely disabled it, or you’re using some odd flavor of bash/csh that hs it disabled by default. More info here: https://www.gnu.org/savannah-checkouts/gnu/bash/manual/bash.html#History-Interaction
Also, the background of the reply window I’m typing in is the dar gray of the site and the font color is a dark gray, too, making it near impossible to write this out. I ended up writing this response in a text editor then pasting it. I’ll email you a screenshot.
Daniel - 05/04/2021
Sorry, but history expansion is enabled by default only for interactive shells. Scripts don’t run using interactive shells, so history expansion is disabled and you can perfectly use the ! between double quotes. I never disabled history expansion explicitly and I never had the problem you mentioned after writing hundreds of scripts in many different Linux systems. But of course I’m not saying that there can’t be some linux distro or some specific configuration on some systems that brings history expansion enabled for non-interactive shells. If so, just escape the character as you said and you’re done.
Thanks for your warning about the font color, now it’s fixed!
Zachary - 05/04/2021
Still having the font color issue on this reply form 🙁
I think there is where the disconnect is — you’re running it in non-interactive, while I’m running in interactive. To me, this seems like a run once script to migrate older machines, so I wouldn’t be running it non-interactively. Curious as to your use case that would make non-interactive ideal.
Daniel - 13/04/2021
With non-interactive I mean that it’s run from a script even if it’s launched interactively from the command line. Take a look at this simple example and you will see what I mean:
$ cat hello-world.txt
#! /bin/bash
echo “Hello world!”
$ ./hello-world.txt
Hello world!
$ echo “Hello world!”
-bash: !”: event not found
It’s exactly the same command, but first I run it from a script and it works as expected, and then I run it directly from command line and it fails as you said. But as my article is about a script, the first approach is perfectly valid.
Regards!
jack - 07/09/2022
How do we run this script ? can you please provide steps for that ?
Daniel - 11/09/2022
You save it to a file, ie change-gp2-to-gp3.sh, then run the command chmod u+x change-gp2-to-gp3.sh to give it execution permission, and then run ./change-gp2-to-gp3.sh from the same directory the script is stored to execute it.
Dennis Ogunfiditimi - 06/07/2021
Hi Daniel – How can I add your script to a Lambda function and Run a monthly event to check all the volume in my account and auto modify volume types from a gp2 – gp3 on a Monthly basis? Thank you
Rook Bravo - 31/10/2021
Create a trigger in cloudwatch events or eventbridge and set a schedule or a cron expression to trigger this lambda function.
Victor Chavez - 08/07/2021
no need for jq. use:
–query ‘Volumes[].VolumeId’
Rook Bravo - 27/10/2021
I took the liberty of enhancing this script a little further to Migrate gp2 to gp3 volumes of specific ec2 instances by tags
region=”us-east-1″
for ec2_instance_id in $(aws ec2 describe-instances –region $region –filter “Name=tag:env, Values=dev” | jq -r ‘.Reservations[].Instances[].InstanceId’)
do
for volume_id in $(aws ec2 describe-instances –region $region –instance-id $ec2_instance_id | jq -r ‘.Reservations[].Instances[].BlockDeviceMappings[].Ebs.VolumeId’)
do
volume_type=$(aws ec2 describe-volumes –region $region –volume-id $volume_id | jq -r ‘.Volumes[].VolumeType’)
if [ $volume_type == ‘gp2’ ]; then
echo “Volume id: $volume_id is of type $volume_type and belongs to Instance id: $ec2_instance_id”
result=$(aws ec2 modify-volume –region ${region} –volume-type=gp3 –volume-id ${volume_id} | jq ‘.VolumeModification.ModificationState’ | sed ‘s/”//g’)
if [ $? -eq 0 ] && [ $result == “modifying” ]; then
echo “OK: volume ${volume_id} changed to state modifying”
else
echo “ERROR: could not change volume ${volume_id} type to gp3”
fi
fi
done
done
Daniel - 01/11/2021
Thanks!
Lokesh - 25/03/2022
getting error for thisjq: error: syntax error, unexpected INVALID_CHARACTER, expecting $end (Unix shell quoting issues?) at , line 1:
‘.Reservations[].Instances[].InstanceId’
jq: 1 compile error
‘ascii’ codec can’t decode byte 0xe2 in position 0: ordinal not in range(128)
script .
Ratul Dutta - 02/02/2022
Hi Daniel.. your script helped a lot today . need to convert more than 200 disks from gp2 to gp3.. done easily using your script… Thanks a lot.. 🙂
Daniel - 02/02/2022
Great! Good to know 🙂
Rocket - 03/06/2022
Hi Daniel, this script LGTM OMG!
your script saved me from a lot of gp2 volumes hell. this script works good without any problem.
Chandler - 11/08/2022
Hi Daniel!
Thanks for the awesome script, works like a charm 🙂