Tag: SysAdmin

Amazon EMR Serverless now supports Apache Spark 4.0.1 (preview). With Spark 4.0.1, you can build and maintain data pipelines more easily with ANSI SQL and VARIANT data types, strengthen compliance and governance frameworks with Apache Iceberg v3 table format, and deploy new real-time applications faster with enhanced streaming capabilities. This enables your teams to reduce technical debt and iterate more quickly, while ensuring data accuracy and consistency. With Spark 4.0.1, you can build data pipelines with standard ANSI SQL, making it accessible to a larger set of users who don't know programming languages like Python or Scala. Spark 4.0.1 natively supports JSON and semi-structured data through VARIANT data types, providing flexibility for handling diverse data formats. You can strengthen compliance and governance through Apache Iceberg v3 table format, which provides transaction guarantees and tracks how your data changes over time, creating the audit trails you need f

Amazon SageMaker now supports Amazon Athena for Apache Spark, bringing a new notebook experience and fast serverless Spark experience together within a unified workspace. Now, data engineers, analysts, and data scientists can easily query data, run Python code, develop jobs, train models, visualize data, and work with AI from one place, with no infrastructure to manage and second-level billing. Athena for Apache Spark scales in seconds to support any workload, from interactive queries to petabyte-scale jobs. Athena for Apache Spark now runs on Spark 3.5.6, the same high-performance Spark engine available across AWS, optimized for open table formats including Apache Iceberg and Delta Lake. It brings you new debugging features, real-time monitoring in the Spark UI, and secure interactive cluster communication through Spark Connect. As you use these capabilities to work with your data, Athena for Spark now enforces table-level access controls defined in AWS Lake Formation.

Today, AWS Payments Cryptography announces support for hybrid post-quantum (PQ) TLS to secure API calls. With this launch, customers can future-proof transmissions of sensitive data and commands using ML-KEM post-quantum cryptography. Enterprises operating highly regulated workloads wish to reduce post-quantum risks from “harvest now, decrypt later”. Long-lived data-in-transit can be recorded today, then decrypted in the future when a sufficiently capable quantum computer becomes available. With today’s launch, AWS Payment Cryptography joins data protection services such as AWS Key Management Service (KMS) in addressing this concern by supporting PQ-TLS. To get started, simply ensure that your application depends on a version of AWS SDK or browser that supports PQ-TLS. For detailed guidance by language and platform, visit the PQ-TLS enablement do

AWS Device Farm enables mobile and web developers to test their apps using real mobile devices and desktop browsers. Starting today, you can connect to a fully managed Appium endpoint using only a few lines of code and run interactive tests on multiple physical devices directly from your IDE or local machine. This feature also seamlessly works with third-party tools such as Appium Inspector — both hosted and local versions — for all actions including element inspection. Support for live video and log streaming enables you to get faster test feedback within your local workflow. It complements our existing server-side execution which gives you the scale and control to run secure enterprise-grade workloads. Taken together, Device Farm now offers you the ability to author, inspect,

Amazon EC2 Image Builder now supports automatic versioning for recipes and automatic build version incrementing for components, reducing the overhead of managing versions manually. This enables you to increment versions automatically and dynamically reference the latest compatible versions in your pipelines without manual updates. With automatic versioning, you no longer need to manually track and increment version numbers when creating new versions of your recipes. You can simply place a single 'x' placeholder in any position of the version number, and Image Builder detects the latest existing version and automatically increments that position. For components, Image Builder automatically increments the build version when you create a component with the same name and semantic version. When referencing resources in your configurations, wildcard patterns automatically resolve to the highest available version matching the specified pattern, ensuring your pipelines always use t

AWS announces the launch of natural language test Q&A generation for Automated Reasoning checks in Amazon Bedrock Guardrails. Automated Reasoning checks uses formal verification techniques to validate the accuracy and policy compliance of outputs from generative AI models. Automated Reasoning checks deliver up to 99% accuracy at detecting correct responses from LLMs, giving you provable assurance in detecting AI hallucinations while also assisting with ambiguity detection in model responses. To get started with Automated Reasoning checks, customers create and test Automated Reasoning policies using natural language documents and sample Q&As. Automated Reasoning checks generates up to N test Q&As for each policy using content from the input document, reducing the work required to go from initial policy generation to production-ready, refined policy. Test generation for Automated Reasoning checks is now available in the US (N. Virginia), US (Ohio), US (O

AWS IoT Core now supports a SET clause in IoT rules-SQL, which lets you set and reuse variables across SQL statements. This new feature provides a simpler SQL experience and ensures consistent content when variables are used multiple times. Additionally, a new get_or_default() function provides improved failure handling by returning default values while encountering data encoding or external dependency issues, ensuring IoT rules continue execution successfully. AWS IoT Core is a fully managed service that securely connects millions of IoT devices to the AWS cloud. Rules for AWS IoT is a component of AWS IoT Core which enables you to filter, process, and decode IoT device data using SQL-like statements, and route the data to 20+ AWS and third-party services.

Amazon EMR 7.12 is now available featuring the new Apache Iceberg v3 table format with Apache Iceberg 1.10. This release enables you to reduce costs when deleting data, strengthen governance and compliance through better tracking for row level changes, and enhance data security with more granular data access control. With Iceberg v3, you can delete data cost-effectively because Iceberg v3 marks deleted rows without rewriting entire files - speeding up your data pipelines while reducing storage costs. You get better governance and compliance capabilities through automatic tracking of every row’s creation and modification history, creating the audit trails needed for regulatory requirements and change data capture. You can enhance data security with table-level encryption, helping you meet privacy regulations for your most sensitive data. With Apache Spark 3.5.6 included in this release, you can leverage these Iceberg 1.10 capabilities for building robust data lake

Amazon Aurora DSQL now supports a maximum storage limit of 256 TiB, doubling the previous limit of 128 TiB. Now, customers can store and manage larger datasets within a single database cluster, simplifying data management for large-scale applications. With Aurora DSQL, customers only pay for the storage they use and storage automatically scales with usage, ensuring that customers do not need to provision storage upfront. All Aurora DSQL clusters by default have a storage limit of 10 TiB. Customers that desire clusters with higher storage limits can request a limit increase using either the Service Quotas console or AWS CLI. Visit the

Today, we're excited to announce the addition of Web Bot Auth (WBA) support in AWS WAF, providing a secure and standardized way to authenticate legitimate AI agents and automated tools accessing web applications. This new capability helps distinguish trusted bot traffic from potentially harmful automated access attempts. Web Bot Auth is an authentication method that leverages cryptographic signatures in HTTP messages to verifythat a request comes from an automated bot. Web Bot Auth is used as a verification method for verified bots and signed agents. It relies on two active IETF drafts: a directory draft allowing the crawler to share their public keys, and a protocol draft defining how these keys should be used to attach crawler's identity to HTTP requests. AWS WAF now automatically allows verified AI agent traffic Verified WBA bots will now be automatically allowed by default, previously Category AI blocked unverified bots, this behavior is now refined to respect WB