I was locally testing a PySpark script I’d written for an AWS Glue job I was building when I ran across the error relating to the call to getResolvedOptions(). The call was generating a KeyLookup exception. The problem was with the argv parameter I supplied.

When a Glue job is executes, parameters are passed to the script through sys.argv. Typically, you pass sys.argv to getResolvedOption(args, options) with the options you want to tease from the list – see Accessing Parameters Using getResolvedOptions for details.

You can mimic this behavior when running this script locally:

from pprint import pprint as pp
from awsglue.utils import getResolvedOptions

argv = ['whatevs', '--JOB_NAME=ThisIsMySickJobName']
args = getResolvedOptions(argv, ['JOB_NAME'])

pp(args)

The following is me running a script that contains the above code locally:

The trick is that the list passed as the argv parameter needs values to use the pattern:

--KEY=VALUE

For example…

--JOB_NAME=ThisIsMySickJobName