1.https://docs.datadoghq.com/serverless/installation/nodejs/?tab=datadogcli:
The preceding approach is a Datadog cli approach in which I followed the steps below:
As a prerequisite, Datadog-cli must be installed.
Using the Datadog-cli interactive approach, enter the required values such as the AWS Secret Key, AWS Access Key, Datadog Api key, and lambda function name.
This will add the layers required for forwarding logs to Datadog to the lambda function.
This would last for 40 minutes because we would need to supply necessary AWS credentials such as AWS access ID, AWS secret key, and Datadog Api Key, and we would need to login to AWS from a terminal using our id
Datadog Cli approach is manually login-in into AWS from the console and entering all the credentials and running the manual commands which is only suitable to get logs of lambda functions for time being, it’s not at all suggested in a long run, because the credentials of AWS, access id, secret key and also the token changes every 40 mins due to my organization internal tool.
2. Custom approach :https://docs.datadoghq.com/serverless/installation/nodejs/?tab=custom
the approach involves
- Add the Datadog related layers to the lambda function from the AWS console
- Update the handler of the lambda function to use the Datadog handler
- Update the Environment variables (DD_API_KEY, DD_SITE, DD_LAMBDA_HANDLER)
this approach is actually recommend in organization most of the teams follow this approach, straight forward, a bit manual of adding the layers to the existing lambda function but we can automate the approach using the terraform, it’s not that expensive (try to find out the price), doesn’t involve running any commands on the terminal like Datadog CLI method. Use more stuff and compare to create the P5 document. We would be able to get Datadog Support if something goes wrong
3.Kinesis firehose Approach : Firehose approach is not used by any teams in my organization , try to figure out the cost and compare it with the alternatives
https://www.datadoghq.com/blog/stream-logs-with-kinesis-firehose-and-datadog/
kinesis firehose approach does’nt have an out of the box solution for taking logs from s3 buckets. The Datadog Forwarder has a simple template to setup and configure to Cloudwatch log Groups and S3c buckets for AWS service logs.
The kinesis Firehose is more efficient at handling the extremely high volume logs
the forwatrder is itself a Lambda function that is triggered on Cloud-watch log groups and S3 buckets and forwards data from these to datadog. If you use the Lambda library when a lambda function is invoked it will write details about that process to a log group, and the forwarder can prase this and send the relavent logs, metrics and traces to datadog.
TRY to read the approve three approches i have mentioned and create a P5 document and mostly try to lean towards custom approach ( 2nd approach ) which I have described.
p5 document example:
1. Problem Statement
What deployment strategy for PingAuthorize policies in staging and production environment should we use?
2. Alternatives
We have identified the following alternatives:
| Alternative | Details |
|---|---|
| 1. Pipeline copies header file and deployment package from N-1 env |
|
| 2. Pipeline sync the s3 buckets |
|
| 3. Use a repository and treat as policy release |
|
| 4. Get deployment package from PAP and generate header file without Digest Footer |
|
| 5. Push deployment package from PAP to a temp bucket and have a pipeline with approval process to move the package to the target bucket. |
|
Note: DP refers to Deployment Package which contains the policies to be deployed.
REJECTED ALTERNATIVES
Before evaluation, the following alternatives were proposed, but rejected:
| Alternative | Rejection Rationale |
|---|---|
| Use PAP to deploy to staging and production environment | Approvals process on PingAuthorize PAP is not elaborate enough for our use cases and automation for self-service is complex |
3. Selection Criteria
| Criteria | Details | Priority | Weight |
|---|---|---|---|
| Operational Excellence: Support rollback strategy | Ability to roll back from the deployed policies | 1 | 10 |
| Reliability: Desired policy is deployed | Ensures that the targeted policy is deployed in the proper environment | 2 | 10 |
| Operational Excellence: Support working on multiple policy releases | Allow multiple policy developers of the same team to work on multiple releases simultaneously | 3 | 10 |
| Reliability: Risk of incompatibility with future’s Ping upgrade | Assess potential incompatibility with future’s Ping upgrade | 4 | 10 |
| Performance Efficiency: Support max history clean up | Allow clean-up of policy deployment package through max history clean up | 5 | 5 |
| Timeline: How fast is the implementation | How much effort is required to implement | 6 | 5 |
| Ping Deployment Pattern Recommendation |
Similarity to Ping’s Recommendation
|
7 | 5 |
REJECTED CRITERIA
The following criteria that would normally be part of most CIAM P5 evaluation criteria have not been considered:
| Criteria | Rationale for rejecting |
|---|---|
| [Reject Criteria 1] | [Rationale] |
| [Reject Criteria 2] | [Rationale] |
4. Evaluation
We will evaluate the Alternatives against the Selection Criteria below. For each combination of alternative and criteria, we will list how that alternative impacts the criteria.
| Alternative | Operational Excellence: Support rollback strategy | Reliability: Desired policy is deployed | Operational Excellence: Support working on multiple policy releases | Reliability: Risk of incompatibility with future’s Ping upgrade | Performance Efficiency: Support max history clean up | Timeline: How fast is the implementation | Ping Deployment Pattern Recommendation | Score |
|---|---|---|---|---|---|---|---|---|
| 1. Pipeline copies header file and deployment package from N-1 env | support rollback up to N-1 version | only current policy from N-1 env can be deployed (assumption it is desired policy) | N+1 release cannot be tested prior to N is deployed in prod | Low risk as it is using Ping’s generated files | Support max history clean up in the pipeline | medium complexity to implement | Recommended by Ping at some point | 41/55 |
| 5/10 | 5/10 | 8/10 | 10/10 | 5/5 | 3/5 | 5/5 | ||
| 2. Pipeline sync the s3 buckets | support rollback up to N-1 version | only current policy from N-1 env can be deployed (assumption it is desired policy) | N+1 release cannot be tested prior to N is deployed in prod | Low risk as it is using Ping’s generated files | Does not support max history clean up in the pipeline | Easy to implement | Similar to Ping’s recommendation | 36/55 |
| 5/10 | 5/10 | 8/10 | 10/10 | 0/5 | 5/5 | 3/5 | ||
| 3. Use a repository and treat as policy release | support rollback up to max history version | desired policy is guaranteed to be deployed by using DP id | flexibility in staging allows N+X releases to be tested prior to N is deployed in prod | Low risk as it is using Ping’s generated files | Support max history clean up in the pipeline | High complexity to implement due to new connectivity with Artifactory | Differs from Ping recommendation in which PAP was the repository | 45/55 |
| 10/10 | 10/10 | 10/10 | 10/10 | 5/5 | 0/5 | 0/5 | ||
| 4. Get deployment package from PAP and generate header file without Digest Footer | support rollback up to max history version | desired policy is guaranteed to be deployed by using DP id | flexibility in staging allows N+X releases to be tested prior to N is deployed in prod | Medium risk as it is generating the header file | Support max history clean up in the pipeline | medium complexity to implement | Similar to Ping’s recommendation | 46/55 |
| 10/10 | 10/10 | 10/10 | 5/10 | 5/5 | 3/5 | 3/5 | ||
| 5. Push deployment package from PAP to a temp bucket and have a pipeline with approval process to move the package to the target bucket. | support rollback up to max history version | desired policy is guaranteed to be deployed by using deployment package version name. | flexibility in staging allows N+X releases to be tested prior to N is deployed in prod | Low risk as it is using Ping’s generated files | Support max history clean up in the pipeline | Easy to implement | Similar to Ping’s recommendation | 53/55 |
| 10/10 | 10/10 | 10/10 | 10/10 | 5/5 | 5/5 | 3/5 |
5. Recommendation:
After discussion with Platform Engineering Architect, the use of artifactory was denied. Further options such as using S3 or Github releases as repository were explored. Afterwards, we came up with Option 5 and agreed to continue with it.
