Skip to main content

CLI data fetching

Fetcher is the module that fetches data from the cloud providers. The aim is to create a unified data structure for data from different cloud providers through a decoupled architecture.

Using the fetcher, one does not need a running server to fetch data from the cloud providers.

CLI​

The fetcher can be used as a CLI tool to fetch data from the cloud providers. There are three ways to use the fetcher as a CLI tool:

  1. fetcher - Provide a JSON file of the specified format and output JSON
  2. fetch - input flags to specify cloud provider, credentials taken from cloud CLI tool configuration
  3. localk8s - fetch data from a kubernetes cluster by providing a kubeconfig file
tip

Output format is the same for all methods, results can be uploaded to SCCCE web for further analysis.

Input Format (Fetcher)​

When fetching data using sccce fetcher, you will need to provide a JSON file that specifies the project, credentials, and rulesets to use for the fetch. The input format is shown in Fetcher JSON Input.

sccce fetcher --project project.json
tip

If sccce is downloaded from the GitHub release, a project.json file is created in the current directory (which is where fetcher reads by default). You can simply edit this file to specify the project, credentials, and rulesets to use for the fetch.

If you want to specify the project.json file in a different directory, you can use the --project flag (short -p) to specify the path to the project.json file.

Input Format (Fetch)​

When fetching data using sccce fetch, you can only fetch for one cloud provider at a time.

sccce fetch <provider> -s <service1,service2> -r <region1,region2>
tip

The credentials are taken from the cloud CLI tool configuration, so make sure you have the correct credentials for the cloud provider.

Input Format (Localk8s)​

When fetching data using sccce localk8s, you will need to provide a kubeconfig file. By default the KUBECONFIG environment variable will be used to find the kubeconfig file.

sccce localk8s --kubeconfig kubeconfig.yaml
tip

If you want to specify the kubeconfig file in a different directory, you can use the --kubeconfig flag (short -k) to specify the path to the kubeconfig file.

Output Format​

The fetcher will output a JSON object that contains the data from the cloud providers.

The structure of the JSON object is shown in Fetcher JSON Output.

tip

By default, the output is written to the output/ directory in the current directory. You can use the --output flag to specify a different path for the output file.

For each project, a separate folder is created in the output/ directory, with a unique name. The folder could contain the following files:

  • fetchResults.json - The output JSON file for fetcher
  • fetchResults-<service>.json - The output JSON file for fetch
  • sccce_localk8s_<timestamp>.json - The output JSON file for localk8s
  • Results from third party tools (e.g. prowler)

Third Party Tools​

When running some of the above commands, third party tools can be run to analyze the fetched data.

  • ELK stack - used when elasticsearch is found locally, and the user chooses to save the fetched logs to Elasticsearch (available for AWS CloudWatchLogs)
  • prowler - used when localk8s is run
tip

I have ELK stack running, but I have not been prompted to save fetched logs to Elasticsearch.

  1. The results do not contain any logs, check the fetchResults.json file to see if the logs are present.
  2. Check if the config/app.yml file is set correctly.
  3. Run sccce check to check if the ELK stack is running correctly.