Carlos Arguelles

Methods, systems, and computer-readable media for analysis of code coverage differences across environments are disclosed. A code coverage profiling system determines a first code coverage profile associated with execution of program code in a first environment. The first code coverage profile indicates one or more portions of the program code that were executed in the first environment. The code coverage profiling system determines a second code coverage profile associated with execution of… Show more

Methods, systems, and computer-readable media for analysis of code coverage differences across environments are disclosed. A code coverage profiling system determines a first code coverage profile associated with execution of program code in a first environment. The first code coverage profile indicates one or more portions of the program code that were executed in the first environment. The code coverage profiling system determines a second code coverage profile associated with execution of the program code in a second environment. The second code coverage profile indicates one or more portions of the program code that were executed in the second environment. The code coverage profiling system performs a comparison of the first code coverage profile and the second code coverage profile. The comparison determines a difference between the portions of the program code that were executed in the first and second environments. Show less

Methods, systems, and computer-readable media for distributed canary testing with test artifact caching are disclosed. Using one or more storage components, a test client stores one or more software artifacts for testing of a software product. The client initiates a first test of the software product using the software artifacts) stored in the storage components). In the first test, the client sends a first set of
requests to the software product at a first point in time. The client… Show more

Methods, systems, and computer-readable media for distributed canary testing with test artifact caching are disclosed. Using one or more storage components, a test client stores one or more software artifacts for testing of a software product. The client initiates a first test of the software product using the software artifacts) stored in the storage components). In the first test, the client sends a first set of
requests to the software product at a first point in time. The client initiates a second test of the software product using the software artifacts) stored in the storage components). In the second test, the client sends a second set of requests to the software product at a second point in time. The software artifacts) are maintained in the storage components) between the first point in time and the second point in time. Show less

A test system includes a controller that creates test jobs to simulate a load for testing a target system or service. The jobs specify test data from a data store. The controller adds the test jobs to a job queue at a rate. Workers access jobs from the job queue and request data from the data store as specified in each job and send the request data to the service. Approximations (e.g., metric values indicating performance of the worker host resources, or performance of the system under test or… Show more

A test system includes a controller that creates test jobs to simulate a load for testing a target system or service. The jobs specify test data from a data store. The controller adds the test jobs to a job queue at a rate. Workers access jobs from the job queue and request data from the data store as specified in each job and send the request data to the service. Approximations (e.g., metric values indicating performance of the worker host resources, or performance of the system under test or performance of resources that implement the system under test, other feedback, etc.) of actual system characteristics such as the actual number of workers are obtained. Based on the metric values, the controller modu- lates various characteristics of the load generation system, such as a rate at which jobs are added to the job queue or the number of workers processing jobs. Show less

A contributed test management system receives a first request from a consumer system, where the first request comprises a request for a contributed test to be added to a deployment pipeline of a producer system, and where the contributed test is associated with an application component in the deployment pipeline. The contributed test management system causes the contributed test to test a code update provided by the producer system for the application component in the deployment pipeline… Show more

A contributed test management system receives a first request from a consumer system, where the first request comprises a request for a contributed test to be added to a deployment pipeline of a producer system, and where the contributed test is associated with an application component in the deployment pipeline. The contributed test management system causes the contributed test to test a code update provided by the producer system for the application component in the deployment pipeline, detects whether the first test fails during execution, and, if so indicates to the consumer system that the first test has failed. Show less

Methods, systems, and computer-readable media for ordered test execution based on code coverage are disclosed. A suite of tests are executed on a first version of program code to generate data indicative of code coverage of respective tests with respect to the program code. A mapping of the tests to the program code is determined based at least in part on the data indicative of code coverage and is stored. The mapping comprises data indicative of one or more portions of the program code… Show more

Methods, systems, and computer-readable media for ordered test execution based on code coverage are disclosed. A suite of tests are executed on a first version of program code to generate data indicative of code coverage of respective tests with respect to the program code. A mapping of the tests to the program code is determined based at least in part on the data indicative of code coverage and is stored. The mapping comprises data indicative of one or more portions of the program code exercised by respective tests from the suite. Based at least in part on the mapping of the tests to the program code and on data indicative of one or more modified or new portions of a second version of the program code, a subset of the tests is determined that are likely to be exercised by the second version of the program code. Show less

Optimization preferences are defined for optimizing execution of a distributed application. Candidate sets of application parameter values may be tested in test execution environments. Measures of performance for metrics of interest are determined based upon the execution of the distributed application using the candidate sets of application parameter values. Utility curves may be utilized to compute measures of effectiveness for metrics of interest. A multi-attribute rollup operation may… Show more

Optimization preferences are defined for optimizing execution of a distributed application. Candidate sets of application parameter values may be tested in test execution environments. Measures of performance for metrics of interest are determined based upon the execution of the distributed application using the candidate sets of application parameter values. Utility curves may be utilized to compute measures of effectiveness for metrics of interest. A multi-attribute rollup operation may utilize the computed measures of effectiveness and weights to compute a grand measure of merit (MOM) for the candidate sets of application parameter values. An optimized set of application parameter values may then be selected based upon the computed grand MOMs. The optimized set of application parameter values may be deployed to a production execution environment executing the distributed application. Production safe application parameters might also be identified and utilized to optimize execution of the distributed application in a production execution environment. Show less

Methods, systems, and computer-readable media for determining the maximum throughput of a service are disclosed. A first sequence of load tests is initiated for a service host. Individual ones of the load tests comprise determining a respective throughput at the service host for a respective number of concurrent connections to the service host. The number of concurrent connections increases nonlinearly in at least a portion of the first sequence of load tests. The first sequence of load tests… Show more

Methods, systems, and computer-readable media for determining the maximum throughput of a service are disclosed. A first sequence of load tests is initiated for a service host. Individual ones of the load tests comprise determining a respective throughput at the service host for a respective number of concurrent connections to the service host. The number of concurrent connections increases nonlinearly in at least a portion of the first sequence of load tests. The first sequence of load tests is discontinued when the throughput is not increased by a threshold from one of the load tests to the next. An estimated maximum throughput for the service host is determined based at least in part on the first sequence of load tests. The estimated maximum throughput corresponds to a particular number of concurrent connections to the service host. Show less

This patent relates to stress testing a computer system using a non-deterministic technique (e.g., randomly adjusting transaction intensity). Based on a load testing specification, the system: (a) identifies the number of virtual computing instances as well as (b) a particular way of determining a threshold. The threshold that is used to make the request is determined based on historical criteria for the virtual computing service. The historical criteria acts as a source of the… Show more

This patent relates to stress testing a computer system using a non-deterministic technique (e.g., randomly adjusting transaction intensity). Based on a load testing specification, the system: (a) identifies the number of virtual computing instances as well as (b) a particular way of determining a threshold. The threshold that is used to make the request is determined based on historical criteria for the virtual computing service. The historical criteria acts as a source of the non-deterministic character of the non-deterministic testing. For example, the threshold may be determined based on historical pricing data such that the load testing will be performed in a non-deterministic manner with respect to time. Show less

Embodiments presented herein provide techniques for dynamically generating synthetic test data used to test a web service. In one embodiment, a service description language document defining a web service may include a test interface definition. The test interface definition specifies rules for generating the synthetic data to use when testing API interfaces exposed by the web service, e.g., to generate synthetic data needed carry out load and performance testing. Including rules for… Show more

Embodiments presented herein provide techniques for dynamically generating synthetic test data used to test a web service. In one embodiment, a service description language document defining a web service may include a test interface definition. The test interface definition specifies rules for generating the synthetic data to use when testing API interfaces exposed by the web service, e.g., to generate synthetic data needed carry out load and performance testing. Including rules for generating test data in the service description language document provides a centralized and authoritative source for both building and testing the web service. Show less

Techniques, including systems and methods, for testing systems utilize a decoupled architecture. A device for administering a test on a system is decoupled from one or more other devices that apply a load on the system as part of the test. The one or more devices that apply the load on the system and the system itself may be located in a common facility. The one or more devices that apply the load on the system may apply the load in a manner specific to one or more applications executing on the… Show more

Techniques, including systems and methods, for testing systems utilize a decoupled architecture. A device for administering a test on a system is decoupled from one or more other devices that apply a load on the system as part of the test. The one or more devices that apply the load on the system and the system itself may be located in a common facility. The one or more devices that apply the load on the system may apply the load in a manner specific to one or more applications executing on the system. Show less

Methods and systems for performance testing in a software deployment pipeline are disclosed. One or more performance tests are automatically performed on a build of a software product in a test environment in response to deploying the build to the test environment. One or more performance metrics are collected based on the performance tests. Based on the performance metrics, the build of the software product is accepted or rejected.

Methods, systems, and computer-readable media for load testing with automated service dependency discovery are disclosed. A request is received to approve load testing for a service. One or more downstream services are identified for the service. The one or more downstream services are identified based at least in part using automated discovery. The availability of the one or more downstream services for load testing is determined. The request is approved or denied based at least in part on the… Show more

Methods, systems, and computer-readable media for load testing with automated service dependency discovery are disclosed. A request is received to approve load testing for a service. One or more downstream services are identified for the service. The one or more downstream services are identified based at least in part using automated discovery. The availability of the one or more downstream services for load testing is determined. The request is approved or denied based at least in part on the availability of the one or more downstream services for load testing. Show less

A generic transaction generator framework for testing a network-based production service may work in conjunction with a product-specific transaction creator module that executes transactions on the service. The transaction creator module may include runtime-discoverable information, such as source code annotations, to communicate product specific details to the framework. Runtime-discoverable information may identify transaction types, transaction methods, as well as dependencies between… Show more

A generic transaction generator framework for testing a network-based production service may work in conjunction with a product-specific transaction creator module that executes transactions on the service. The transaction creator module may include runtime-discoverable information, such as source code annotations, to communicate product specific details to the framework. Runtime-discoverable information may identify transaction types, transaction methods, as well as dependencies between different transaction types and transaction methods. The framework may generate and execute various test transactions and may call a substituted transaction method for a transaction type on which a generated transaction depends prior to calling the generated transaction. The output from the substituted transaction may be used as input to the generated transaction when executed subsequently. Show less

Technologies are disclosed herein for generating comments in a source code review tool using code analysis tools. A producer module can be executed in order to obtain source code from a source code review tool. One or more source code analysis modules can then be executed in order to analyze the source code. A reporter module can then store the output of the source code analysis modules as comments in the source code review tool for use by a developer of the source code. The producer, reporter,… Show more

Technologies are disclosed herein for generating comments in a source code review tool using code analysis tools. A producer module can be executed in order to obtain source code from a source code review tool. One or more source code analysis modules can then be executed in order to analyze the source code. A reporter module can then store the output of the source code analysis modules as comments in the source code review tool for use by a developer of the source code. The producer, reporter, and source code analysis modules can be executed in response to a request from the source code developer to perform a source code review, by a job scheduler, or in another manner. An application programming interface (API) exposed by the source code review tool can be utilized to obtain the source code and to store the comments associated with the source code. Show less

Embodiments presented herein provide techniques for evaluating an asynchronous application using a test framework. The test framework may perform a load test of an asynchronous application or service composed from a collection of applications or services. To do so, the test framework may submit transactions to a distributed application at a specified transaction rate and monitor how the distributed application operates at that transaction rate. An aggregate load test component may evaluate the… Show more

Embodiments presented herein provide techniques for evaluating an asynchronous application using a test framework. The test framework may perform a load test of an asynchronous application or service composed from a collection of applications or services. To do so, the test framework may submit transactions to a distributed application at a specified transaction rate and monitor how the distributed application operates at that transaction rate. An aggregate load test component may evaluate the remaining work pending at work accumulation points of the distributed application to determine whether the distributed application can sustain the specified transaction rate. A transaction tracking component may initiate transactions to generate load at the specified transaction rate without blocking while the transactions are processed by the distributed application. Show less

Techniques are described for aggregating code coverage data generated from various types of testing of software modules, and automatically determining whether to promote software upwards in a multi-level software deployment hierarchy based on the aggregated code coverage data. In embodiments, a code coverage metric is determined for a software module, and the metric is compared to a set of promotion criteria, including whether the metric meets a predetermined threshold for quality. In some… Show more

Techniques are described for aggregating code coverage data generated from various types of testing of software modules, and automatically determining whether to promote software upwards in a multi-level software deployment hierarchy based on the aggregated code coverage data. In embodiments, a code coverage metric is determined for a software module, and the metric is compared to a set of promotion criteria, including whether the metric meets a predetermined threshold for quality. In some cases, the threshold may be a general threshold, a threshold based on the level of possible promotion, and/or a threshold that is based on an identified category for the software module such as whether the module is a front-end module, a shared module, a legacy module, or a critical module. Show less

A software testing framework provides functionality for utilizing pre-existing tests to load and performance test a network service. Methods can be tagged with annotations indicating that they are tests, such as integration tests. The methods implementing the integration tests can also be tagged with other types of annotations that can be used to select individual tests for use in testing, such as annotations indicating whether a test is a positive or negative test, annotations specifying… Show more

A software testing framework provides functionality for utilizing pre-existing tests to load and performance test a network service. Methods can be tagged with annotations indicating that they are tests, such as integration tests. The methods implementing the integration tests can also be tagged with other types of annotations that can be used to select individual tests for use in testing, such as annotations indicating whether a test is a positive or negative test, annotations specifying dependencies upon other tests, or annotations indicating that a test is a member of a test suite. The annotations can be utilized in conjunction with test selection criteria to select individual integration tests for use in load and performance testing of the network service. The selected integration tests can be deployed to and executed upon load-generating instances to execute the integration tests and generate requests to the network service at high throughput. Show less

A generic transaction generator framework for testing a network-based production service may work in conjunction with a product-specific transaction creator module that executes transactions on the service. The transaction creator module may include runtime-discoverable information, such as source code annotations, to communicate product specific details to the framework. Runtime-discoverable information may identify transaction types, transaction methods, data provider methods and data… Show more

A generic transaction generator framework for testing a network-based production service may work in conjunction with a product-specific transaction creator module that executes transactions on the service. The transaction creator module may include runtime-discoverable information, such as source code annotations, to communicate product specific details to the framework. Runtime-discoverable information may identify transaction types, transaction methods, data provider methods and data sources. The framework may generate and execute various test transactions and may call a data provider method to prepare data for the transaction and pass the prepared data to a transaction method. The framework may also load and parse test data from a data source and provide the test data to the data provider method for use when preparing data for the transaction. Show less

Technologies are disclosed herein for distributed software testing. A software testing service is disclosed that can receive tests for software under test. The software testing service can determine a number of test instances to utilize to perform the tests. The determined number of test instances can then be provisioned, such as through the use of an on-demand computing service executing in a service provider network. The test instances can also be configured with the software under test and a… Show more

Technologies are disclosed herein for distributed software testing. A software testing service is disclosed that can receive tests for software under test. The software testing service can determine a number of test instances to utilize to perform the tests. The determined number of test instances can then be provisioned, such as through the use of an on-demand computing service executing in a service provider network. The test instances can also be configured with the software under test and a test runner capable of performing the tests on the software. The software testing service can place the tests on a queue, such as a queue provided by a queue service. The test runners executing on the test instances can dequeue the tests and perform the tests on the software. Once the testing of the software under test has been completed, the test instances can be de-provisioned. Show less

Optimization preferences are defined for optimizing execution of a distributed application. Candidate sets of application parameter values may be tested in test execution environments. Measures of performance for metrics of interest are determined based upon the execution of the distributed application using the candidate sets of application parameter values. Utility curves may be utilized to compute measures of effectiveness for metrics of interest. A multi-attribute rollup operation may… Show more

Optimization preferences are defined for optimizing execution of a distributed application. Candidate sets of application parameter values may be tested in test execution environments. Measures of performance for metrics of interest are determined based upon the execution of the distributed application using the candidate sets of application parameter values. Utility curves may be utilized to compute measures of effectiveness for metrics of interest. A multi-attribute rollup operation may utilize the computed measures of effectiveness and weights to compute a grand measure of merit (MOM) for the candidate sets of application parameter values. An optimized set of application parameter values may then be selected based upon the computed grand MOMs. The optimized set of application parameter values may be deployed to a production execution environment executing the distributed application. Show less

Methods, systems, and computer-readable media for determining the maximum throughput of a service are disclosed. A first sequence of load tests is initiated for a service host. Individual ones of the load tests comprise determining a respective throughput at the service host for a respective number of concurrent connections to the service host. The number of concurrent connections increases nonlinearly in at least a portion of the first sequence of load tests. The first sequence of load tests… Show more

Methods, systems, and computer-readable media for determining the maximum throughput of a service are disclosed. A first sequence of load tests is initiated for a service host. Individual ones of the load tests comprise determining a respective throughput at the service host for a respective number of concurrent connections to the service host. The number of concurrent connections increases nonlinearly in at least a portion of the first sequence of load tests. The first sequence of load tests is discontinued when the throughput is not increased by a threshold from one of the load tests to the next. An estimated maximum throughput for the service host is determined based at least in part on the first sequence of load tests. The estimated maximum throughput corresponds to a particular number of concurrent connections to the service host. Show less

This patent relates to end-to-end testing of long running asynchronous data processing systems where it may take hours or days for data to get from one end of the system to the other (e.g., a large-scale ecommerce website in which data is logged an analyzed). The invention simplifies end-to-end testing of such systems by dividing the testing into independent single-stage tests that can be inter-connected to form longer running tests. A scalable test data repository service handles the storing… Show more

This patent relates to end-to-end testing of long running asynchronous data processing systems where it may take hours or days for data to get from one end of the system to the other (e.g., a large-scale ecommerce website in which data is logged an analyzed). The invention simplifies end-to-end testing of such systems by dividing the testing into independent single-stage tests that can be inter-connected to form longer running tests. A scalable test data repository service handles the storing and fetching of actual and expected data across test machines, abstracting the actual storage method so that the tests do not have to be storage-dependent and can easily be migrated to other storage solutions if needed. With this infrastructure, end-to-end integration tests can be performed for long running multi-stage data processing service in a fraction of the time required by other test frameworks. Show less

This patent relates to helping ensure production systems are robust by allowing a user to create a load generator in Java frameworks to test their services before they are moved to production. The invention uses a generic transaction generator that works together with a product-specific transaction creator to generate test transactions to load the service. The transaction creator includes runtime-discoverable information, such as source code annotations, to communicate product specific details… Show more

This patent relates to helping ensure production systems are robust by allowing a user to create a load generator in Java frameworks to test their services before they are moved to production. The invention uses a generic transaction generator that works together with a product-specific transaction creator to generate test transactions to load the service. The transaction creator includes runtime-discoverable information, such as source code annotations, to communicate product specific details to the transaction generator. The runtime-discoverable information may identify initialization methods, terminate methods, transaction types, transaction methods, as well as dependencies between different transactions. Additionally, the runtime-discoverable information can identify various testing parameters, such as transaction rate, testing period and/or a desired distribution among various transaction types. The transaction generator generates various test transactions, causes them to be executed on the service, and collects performance metrics regarding how well the service performed the test transactions. Show less

This patent relates to speeding up software development through the use of ordered test execution to enable faster feedback. In a test suite for testing a computer program, the individual tests are ordered in a sequence based on their likelihood of failure. The tests that are more likely to fail are performed earlier, so that a developer can be notified of failure or the test suite can be suspended as soon as possible.

This patent relates to a highly-scalable test system for testing large network-based production services (e.g., Amazon.com). The test system includes scalable job controllers for specifying test jobs, and a scalable pool of workers to perform the test jobs on the production service. A job queue is used to de-couple the job controllers from the workers such that the job controllers and the workers may be scaled independently from one another. The test jobs may utilize previously captured… Show more

This patent relates to a highly-scalable test system for testing large network-based production services (e.g., Amazon.com). The test system includes scalable job controllers for specifying test jobs, and a scalable pool of workers to perform the test jobs on the production service. A job queue is used to de-couple the job controllers from the workers such that the job controllers and the workers may be scaled independently from one another. The test jobs may utilize previously captured production request data from the production service to create a test load. The test system also includes an auto-scaling feature to scale the workers as needed (and independently from the job controllers). Show less

This patent relates to helping ensure production systems are robust by modeling potential situations (e.g., what if right now, 30% of the transactions are X, but this will go up to 80% in the future?) with real world production data instead of randomly generated data. This is accomplished by clustering real world production data for use in generating a predictive test load. Real production data is turned into vectors and the vectors are clustered. The clusters form patterns that are useful in… Show more

This patent relates to helping ensure production systems are robust by modeling potential situations (e.g., what if right now, 30% of the transactions are X, but this will go up to 80% in the future?) with real world production data instead of randomly generated data. This is accomplished by clustering real world production data for use in generating a predictive test load. Real production data is turned into vectors and the vectors are clustered. The clusters form patterns that are useful in the creation of predicted data loads that can be used to test a production system. In particular, the cluster definitions are used to sort the production data into a predictive test load. Show less

This patent relates to improving the scalability of load testing systems. A total target load is divided into smaller jobs, and the jobs are placed in a queue. Individual worker hosts retrieve the jobs from the queue and execute them locally to implement the load test. The individual worker hosts self-adjust to different portions of the total target load, and the number of worker hosts may be automatically adjusted according to the requirements of the load test.

This patent relates to a highly-scalable test system for testing large network-based production services (e.g., Amazon.com). The test system includes scalable job controllers for specifying test jobs, and a scalable pool of workers to perform the test jobs on the production service. A job queue is used to de-couple the job controllers from the workers such that the job controllers and the workers may be scaled independently from one another. The test jobs may utilize previously captured… Show more

This patent relates to a highly-scalable test system for testing large network-based production services (e.g., Amazon.com). The test system includes scalable job controllers for specifying test jobs, and a scalable pool of workers to perform the test jobs on the production service. A job queue is used to de-couple the job controllers from the workers such that the job controllers and the workers may be scaled independently from one another. The test jobs may utilize previously captured production request data from the production service to create a test load. The test system also includes an auto-shutdown feature to prevent a test from overly-stressing the production service. In some cases, auto-shutdown may be implemented by killing the job queue. Show less

This patent relates to improving the customer experience by using a test system to ensure that a production service with many hosts can scale to handle abnormal loads. The test system includes scalable job controllers for specifying test jobs, and a scalable pool of workers to perform the test jobs on the production service. A job queue is used to de-couple the job controllers from the workers such that the job controllers and the workers may be scaled independently from one another. The test… Show more

This patent relates to improving the customer experience by using a test system to ensure that a production service with many hosts can scale to handle abnormal loads. The test system includes scalable job controllers for specifying test jobs, and a scalable pool of workers to perform the test jobs on the production service. A job queue is used to de-couple the job controllers from the workers such that the job controllers and the workers may be scaled independently from one another. The test jobs utilize previously captured production request data from the production service to create a test load. Show less

Systems and associated processes for testing a reverse proxy server are disclosed. A backend proxy server test system can receive a request from a reverse proxy server under test. The request may be generated in response to a request from a test client to access a backend service. In responding to the received request, the backend proxy server test system can include a copy of the received request. Upon the test client receiving the response from the proxy server to the test client's request… Show more

Systems and associated processes for testing a reverse proxy server are disclosed. A backend proxy server test system can receive a request from a reverse proxy server under test. The request may be generated in response to a request from a test client to access a backend service. In responding to the received request, the backend proxy server test system can include a copy of the received request. Upon the test client receiving the response from the proxy server to the test client's request, the test client can extract the embedded copy of the received request that the reverse proxy server generated to determine whether it matches the request that a functioning reverse proxy server generates. Based, at least in part on the result of this comparison, the test client can determine whether the reverse proxy server is malfunctioning. Show less

Methods and systems for automated tuning of a service configuration are disclosed. An optimal configuration for a test computer is selected by performing one or more load tests using the test computer for each of a plurality of test configurations. The performance of a plurality of additional test computers configured with the optimal configuration is automatically determined by performing additional load tests using the additional test computers. A plurality of production computers are… Show more

Methods and systems for automated tuning of a service configuration are disclosed. An optimal configuration for a test computer is selected by performing one or more load tests using the test computer for each of a plurality of test configurations. The performance of a plurality of additional test computers configured with the optimal configuration is automatically determined by performing additional load tests using the additional test computers. A plurality of production computers are automatically configured with the optimal configuration if the performance of the additional test computers is improved with the optimal configuration. Show less

Systems and methods for validation of log formats are described herein. Log data is stored via a logging service in a data store or other storage system. An example log or proposed log format is received by the logging service. The proposed log format is validated against validation rules provided by log consumers.

Systems and methods for incrementally building tests and test frameworks may enable cost reduction and/or code sharing between teams testing various products (or features thereof) in different domains or of different product types. Test frameworks may be layered into standalone sub-framework packages that build on each other and whose object classes inherit object classes of lower layers. Sub-framework packages may provide generic utilities, domain-specific testing operations… Show more

Systems and methods for incrementally building tests and test frameworks may enable cost reduction and/or code sharing between teams testing various products (or features thereof) in different domains or of different product types. Test frameworks may be layered into standalone sub-framework packages that build on each other and whose object classes inherit object classes of lower layers. Sub-framework packages may provide generic utilities, domain-specific testing operations, application-specific testing operations, or operations for specific application types (e.g., web services). Test clients may plug into test frameworks at any layer to exercise features of a product under test. Tests may be built from components at multiple levels of abstraction, and may include test operations, test steps that include multiple test operations, and tests that include multiple test steps. Implementation details of products to be tested may only be visible to, and/or exercised by, the lowest-level elements of the tests (e.g., test operations). Show less

One or more computers is configured to run an end-to-end test including at least a plurality of independent tests of multiple stages of an asynchronous multi-stage data processing system. One of the set of independent tests is configured to send a request for test input data from a test data repository service for a particular stage. A converted version of the test input data is obtained. A comparison of the converted version to the output of the particular stage to verify operation of the… Show more

One or more computers is configured to run an end-to-end test including at least a plurality of independent tests of multiple stages of an asynchronous multi-stage data processing system. One of the set of independent tests is configured to send a request for test input data from a test data repository service for a particular stage. A converted version of the test input data is obtained. A comparison of the converted version to the output of the particular stage to verify operation of the particular stage is obtained. The output of the particular stage is transmitted to the test data repository service. One or more computers is configured to provide the test data repository service. The test data repository service is configured to store in the test data storage the output of the particular stage as test input data for a next stage of the asynchronous multi-stage data processing system. Show less

A network-based production service is configured to process client requests for the production service via a network, capture production request data defining the requests and store the production request data in a data store. A test system comprising one or more controllers creates test jobs according to a test plan for testing the production service. The test plan creates a test profile for using specified production request data to simulate a load on the production service. Each job created… Show more

A network-based production service is configured to process client requests for the production service via a network, capture production request data defining the requests and store the production request data in a data store. A test system comprising one or more controllers creates test jobs according to a test plan for testing the production service. The test plan creates a test profile for using specified production request data to simulate a load on the production service. Each job created by the test plan specifies a portion of production request data. A job queue receives and queues test jobs from one or more controllers configured to add test jobs to the job queue according to the test plan. Workers access jobs from the job queue and the production request data from the data store as specified in each job and replay the production request data to the production service.
Show less

Systems and associated processes for testing a reverse proxy server are disclosed. A backend proxy server test system can receive a request from a reverse proxy server under test. The request may be generated in response to a request from a test client to access a backend service. In responding to the received request, the backend proxy server test system can include a copy of the received request. Upon the test client receiving the response from the proxy server to the test client's request… Show more

Systems and associated processes for testing a reverse proxy server are disclosed. A backend proxy server test system can receive a request from a reverse proxy server under test. The request may be generated in response to a request from a test client to access a backend service. In responding to the received request, the backend proxy server test system can include a copy of the received request. Upon the test client receiving the response from the proxy server to the test client's request, the test client can extract the embedded copy of the received request that the reverse proxy server generated to determine whether it matches the request that a functioning reverse proxy server generates. Based, at least in part on the result of this comparison, the test client can determine whether the reverse proxy server is malfunctioning. Show less

Proposed updates to systems are evaluated in a manner that is automated and horizontally scalable. Input to a first system is provided to a second system. The first system and second system process the input and each generates output. The output from the first system and second system is analyzed and differences in the output data between the two systems are identified. Analyzing the output may be performed by a fleet of data processing units and the work of analyzing the output may be… Show more

Proposed updates to systems are evaluated in a manner that is automated and horizontally scalable. Input to a first system is provided to a second system. The first system and second system process the input and each generates output. The output from the first system and second system is analyzed and differences in the output data between the two systems are identified. Analyzing the output may be performed by a fleet of data processing units and the work of analyzing the output may be performed such that differences in the output data are traceable to subsystems of the second system that caused the differences. Show less

Techniques are described for aggregating code coverage data generated from various types of testing of software modules, and automatically determining whether to promote software upwards in a multi-level software deployment hierarchy based on the aggregated code coverage data. In embodiments, a code coverage metric is determined for a software module, and the metric is compared to a set of promotion criteria, including whether the metric meets a predetermined threshold for quality. In some… Show more

Techniques are described for aggregating code coverage data generated from various types of testing of software modules, and automatically determining whether to promote software upwards in a multi-level software deployment hierarchy based on the aggregated code coverage data. In embodiments, a code coverage metric is determined for a software module, and the metric is compared to a set of promotion criteria, including whether the metric meets a predetermined threshold for quality. In some cases, the threshold may be a general threshold, a threshold based on the level of possible promotion, and/or a threshold that is based on an identified category for the software module such as whether the module is a front-end module, a shared module, a legacy module, or a critical module. Show less

A network-based production service is configured to process client requests for the production service via a network, capture production request data defining the requests and store the production request data in a data store. A test system comprising one or more controllers creates test jobs according to a test plan for testing the production service. The test plan creates a test profile for using specified production request data to simulate a load on the production service. Each job created… Show more

A network-based production service is configured to process client requests for the production service via a network, capture production request data defining the requests and store the production request data in a data store. A test system comprising one or more controllers creates test jobs according to a test plan for testing the production service. The test plan creates a test profile for using specified production request data to simulate a load on the production service. Each job created by the test plan specifies a portion of production request data. A job queue receives and queues test jobs from one or more controllers configured to add test jobs to the job queue according to the test plan. Workers access jobs from the job queue and the production request data from the data store as specified in each job and replay the production request data to the production service.
Show less

A new method is provided for using distributed computing for computing process testing. The method includes distributing a feature testing tool, computing feature builds, and data sets to client computing systems. The feature testing tool is run on the client computing systems during times when the client computing system is not otherwise in active use. The feature testing tool prompts the computing feature build to process the data sets, and collects results from the processing of the data… Show more

A new method is provided for using distributed computing for computing process testing. The method includes distributing a feature testing tool, computing feature builds, and data sets to client computing systems. The feature testing tool is run on the client computing systems during times when the client computing system is not otherwise in active use. The feature testing tool prompts the computing feature build to process the data sets, and collects results from the processing of the data set, including any erroneous results. The results from processing the data sets on the client computing systems are provided to a managing computing system. The managing computing system may try to reproduce the erroneous results on other client computing systems with alternate hardware-software configurations, to send erroneous results to an investigation queue to isolate their cause, and to report useful information on the erroneous results to developers working on the computing feature. Show less

A computer-implemented method of generating a dynamic corpus includes generating web threads, based upon corresponding sets of words dequeued from a word queue, to obtain web thread resulting URLs. The web thread resulting URLs are enqueued in a URL queue. Multiple text extraction threads are generated, based upon documents downloaded using URLs dequeued from the URL queue, to obtain text files. New words are randomly obtained from the text files, and the randomly obtained words from the text… Show more

A computer-implemented method of generating a dynamic corpus includes generating web threads, based upon corresponding sets of words dequeued from a word queue, to obtain web thread resulting URLs. The web thread resulting URLs are enqueued in a URL queue. Multiple text extraction threads are generated, based upon documents downloaded using URLs dequeued from the URL queue, to obtain text files. New words are randomly obtained from the text files, and the randomly obtained words from the text files are enqueued in the word queue. This process is iteratively performed, resulting in a dynamic corpus. Show less

A processing device and method may provide natural language assistance for digital image indexing. Candidate keywords may be generated based on an entered keyword for a digital image file. The candidate keywords may be synonyms of the entered keyword, or otherwise related to the entered keyword. The candidate keywords may be ranked as possible keywords for the digital image file based on use of the candidate keywords as keywords of other digital image files. Similarity measures of the other… Show more

A processing device and method may provide natural language assistance for digital image indexing. Candidate keywords may be generated based on an entered keyword for a digital image file. The candidate keywords may be synonyms of the entered keyword, or otherwise related to the entered keyword. The candidate keywords may be ranked as possible keywords for the digital image file based on use of the candidate keywords as keywords of other digital image files. Similarity measures of the other digital image files may be determined based on aspects of the other digital image files compared with the corresponding aspects of the digital image file. Rankings of the candidate keywords may be adjusted based on the similarity measures of the other digital image files. At least some of the ranked candidate keywords may be presented. Show less

Photography auto-triage is described. In embodiment(s), at least a portion of each photograph in a group of photographs can be analyzed with triage filters of an auto-triage service to identify deficient photographs. A deficient photograph can be identified and then removed from the group of photographs.