With the introduction of smaller RPU configuration in Redshift Serverless, we no longer need to worry about infrastructure tuning or security risks and can accommodate many small analytics workloads. We already heavily use AWS services and Amazon Redshift Serverless is perfect for us to consolidate the entire analytics workload onto one platform. For these small analytics workloads we often used services from other vendors that required us to introduce security concerns in transferring data. Our analytics workloads are often small since we are running multiple different data exploration and experimental workloads in our growing business as a startup company. Host: event.url, // event.url is without https! just the DNSĬonst req = https.“Schoo encourages people to keep learning new things for a lifetime by offering live video streaming services and online community. The script is just a simple https request: ' at GetAddrInfoReqWrap.onlookup (node:dns:107:26)' But on aws lambda it works for some urls only and for other urls it can't be resolved and I'm getting this error: I have a simple HTTPS request that works well on my local envoiremnt.

* The following is the error I get when trying different options listed above.ĪnalysisException: .ql.metadata.HiveException: : Unable to execute HTTP request: Connect to failed: connect timed out Read from jdbc table via Glue contextĭynamic_frame = glue_context.create_dynamic_frame_from_options(Ĭonnection_type=jdbc_config, # 'postgresql' * And then I think I should be able to pass the connection info to either a Glue or Spark function to read from the table. Jdbc_config = glue_context.extract_jdbc_conf(connection_name=glue_connector) Glue_context = GlueContext(spark_session.sparkContext) Glue_context.spark_(session_name)ĭef get_jdbc_table(spark_session: SparkSession, glue_connector: str, db_table: str, logger) -> DataFrame: Spark_conf = SparkConf().setAll(pairs=list(spark_config_ems())) Glue_context = GlueContext(spark_context)

Spark_context = SparkContext.getOrCreate() The spark session is returned from another util function and then is passed into this function, which is why I am getting the glue context from the spark context instead of the other way around.įrom pyspark.sql import DataFrame, SparkSession * Retrieving information about the connection I think should work using **glue_context.extract_jdbc_conf()**. * Testing a Glue connection is successful.

Redshift rendezvous