Feat/event based instance replacement - Merged all "new" commit from …

…master (#433)

* Allow flavor to be customised (#359)

* fix typo (#360)

* Fix typo (#361)

* UserData wrappers for CloudFormation helper scripts when using Beanstalk (#366)

* Support custom role for cfn-init in Beanstalk UserData

* Wrappers & refactoring

* Docs

* Docs fixes

* More docs fixes

* Docs fixes

* Yet more docs fixes

* Lambda & Kubernetes config

* AutoSpottingElasticBeanstalk managed policy + Rename beanstalk_cfn_init_role to beanstalk_cfn_wrappers

* Update param description

* Kubernetes config fix

* Rename Beanstalk variable + Move test data

* Add missing permission for AutoSpottingBeanstalk role

* Bring Go to 1.13 (#367)

* merge fixes

* Begin - port regional CloudFormation stack changes to the main template so it can also run from a StackSet

* Merge fix

* Fix to template

* Multiple fixes to handle lambda concurrency

As instaces can be launched in concurrency/overlapping we have the
problems related to multiple lambdas acting on the same ASG

* Removed commented code

* Begin working on:

#354 (comment)

* Progress #354

Created Queue in CF template
Begin sending message

* Progress #354

* Progress #354

* Progress #354

* Progress #354

* Progress #354

* * Use inline python lambda to increase ASG

Use it only if AttachInstances method fails for wrong ASG max size.

* * Use inline python lambda to increase ASG

progress

* * Use inline python lambda to increase ASG

progress

* * Improvements on event based instance replacement

- Ported to StackSet deploy mode.

- Fix to "ScalingActivityInProgress" if launched ondemand instance is
terminated before going "inservice":

	Created method "waitForInstanceStatus", that wait, until a max
	retry (5), that an instance belonging to an ASG is in the desired
	status (InService).
	(Sleep time is 5*retry)

- Fix to multiple problems caused by lambda concurrency changing ASG
MaxSize:
	Created another Lambda (LambdaManageASG) in the same region of
	the main one; code python3.7, inline in template.
	Lambda concurrency is set to one.
	Is function is to change ASG MaxSize by the ammount specified.

	Used a "try-catch approach":
	if AttachInstances return error code "ValidationError" and
	string "update the AutoScalingGroup sizes" is present in error
	message, means that we need to increase ASG MaxSize.
	So we execute method changeAutoScalingMaxSize that invoke
	LambdaManageASG.

	If invoke return error "ErrCodeTooManyRequestsException", means
	that multiple Main lambdas are executing LambdaManageASG, we
	sleep for a random interval part of seconds and retry.

	Method attachSpotInstance now return an int that represent the
	ammount of change to the ASG MaxSize, so that in
	swapWithGroupMember we can defer the call to
	changeAutoScalingMaxSize to decrease the ASG MaxSize of the
	previously increased ammount.

	We use "waitForInstanceStatus" in attachSpotInstance before
	returning to be sure that spot instance has been attached and
	are InService before beginning to terminate ondemand one.

* * Improvements on event based instance replacement

- Add logPrefix to better identify lambda actions in case of concurent
executions.
	TE = Spot terminate event
	ST = Instance start event
	SC = Schedule event

TE and ST are followed by instanceID, SC is followed by activation time.

* * Improvements on event based instance replacement

- fix/improvement on Suspending/Resuming Termination process:
suspend/resume is now handled by LambdaManageASG too.

When i successfully suspend termination, i add a tag
"autospotting_suspend_process_by" with value equals to the instanceId
 event that triggered the main lambda.
When i try to resume termination first check if the value of the tag
above equals the current one.
If not, means that another main lambda, different from the one who suspended
it, is trying to resume the process; in that case i do not resume it.

Considered that LambdaManageASG has a concurrent execution limit set to
1 we can have the following cases:

1)
  *) A main lambda suspend the process.
  *) No other lambda suspend it.
  *) Main lambda resume the process.
  *) Another main lambda suspend the process
  *) ....and so on

2)
  *) A main lambda suspend the process.
  *) Before the first one resume the process another one suspend it (so
  replacing the tag value)
  *) First lambda do not resume the process (tag value differ)
  *) Second lambda resume the process

In the wrost scenario of a Lambda dying before resuming the process, it
will be resumed after another one will suspend it.

* * Improvements on event based instance replacement

code cosmetic fix

* * Improvements on event based instance replacement

- for rand seed instead of using time.Now().UnixNano() we build a seed
based on the instanceId that triggered the event.

The seed is build this way:
for every char of instanceId (starting from third char) we get his rune
 "representation" and sum it to previous one.
We use it as a temporary seed and get a random number between 0 and 9.

The final seed is the concatenation of the generated random numbers.

This way we have a seed number (int64) that depend from the instanceId and
of the same lenght.

* * Improvements on event based instance replacement

- no more need to "revert attach/detach order when running on minimum
capacity".
Defer changeAutoScalingMaxSize and use same logic of
swapWithGroupMember.

* * Improvements on event based instance replacement

- Use suspendResumeProcess for schedule replaceOnDemandInstanceWithSpot
too.

We use it as a trick to avoid rare cases of concurrency between
scheduled and event lambdas.
As lambda that handle "suspendResumeProcess" have a concurrency limit of
one, scheduled and event lambdas, if concurrent, will be "time shifted"
by a random value.
This way they will not execute attachSpotInstance at the same time.

In case of scheduled lambda we add the "S" char to the instanceId used for
the randSeed to avoid that it resume process suspended by event lambda.

- Fix in swapWithGroupMember:
defer asg.suspendResumeProcess for resume so that it will be executed
even if function swapWithGroupMember return error.

* * gofmt cosmetic changes

* Fix to rand.Intn parameter to include even 9

* fix terminateRandomSpotInstanceIfHavingEnough

need another condition to avoid terminating valid spot instance in case ASG have minOnDemand > 0

if all ASG instances are in state running and
  Min OnDemand instance equals total ondemand running and
  all instances running equals desired capacity
means that i do not need to terminate a spot instance

need some testing

* fix terminateRandomSpotInstanceIfHavingEnough

fixes

* fix terminateRandomSpotInstanceIfHavingEnough

changed allInstancesRunning to return ondemand instances running too

* Specify/override multiplier for the on-demand price on a group
level

As i.price already have been multiplied by the global value, if
specified, i need first to divide it by the same value and then multiply
it by the multiplier specific to the ASG.

We need to do this for both scheduled and event actions.
For event we act in function belongsToEnabledASG.
For schedule we act in launchSpotReplacement.

Need deep testing...

* Specify/override multiplier for the on-demand price on a group
    level - fix

in loadConfOnDemandPriceMultiplier need to use
a.config.OnDemandPriceMultiplier in place of
a.region.conf.OnDemandPriceMultiplier

this way a.region.conf.OnDemandPriceMultiplier will conserve the
original global value

* Misc fixes to be able to run go tests

* Misc fixes to be able to run go tests - continue

* Misc fixes to be able to run go tests - end for now

* For Test_autoScalingGroup_terminateRandomSpotInstanceIfHavingEnough
added contion:
"spot capacity is correct, skip termination"

* Merge Updating AWS SDK to latest version - [0ee95c8]

* Merge Relicense to OSL-3 - [972bc61]

* Merge Update readme, mention the relicensing to OSL-3 - [9b438dc]

* Merge Fix `make archive` on macOS [c929391]

* Merge Move some logs to debug [fa0c27b]

* Merge Delete old Gopkg files [5504e34]

* Merge Update dependencies [0e14a7b]

* Merge Support spot price buffer percentage of 0 [26ca955]

* Merge Don't segfault when spot instance doesn't belong to ASG [f313c6d]

* Merge Allow specifying GOOS and GOARCH [8d67d6e]

* Merge Use /bin/bash for shell [0eed8cd]

* Merge Ignore terminating spot instances that don't belong to
AutoSpotting (master) and Enable Spot Termination ASG Checking (event) [33a444c]

* Merge Revert Use /bin/bash for shell [6e45440]

* Merge Pass --abort-on-container-exit to docker-compose [8f90fba]

* Merge Rename travisci make targets to ci [2d76a9a]

* Merge Create FUNDING.yml + Update FUNDING.yml [dde6d85,ca81828]

* Merge Use paginated version of DescribeSpotPriceHistory [9a770b3]

* Merge Delete DescribeSecurityGroups mock [3bfd542]

* Merge Move config loading out of main and add tests for it [30a4392]

* Fixes to Merge Move config loading out of main and add tests for it

* Merge Move logs about incompatible instance types to debug [985d675]

* Merge Remove incorrect Makefile conditionals [a01ee26]

* Merge Actually fail the build if gofmt fails [c391d69]

* Merge Add tools to go.mod [8a9a90b]

* Merge No longer enforce the name of the ElasticBeanstalk IAM policy [3313298]

* Merge Added spot premium [ecf31a5]

* Merge Update how bid price is calculated for premium instances [8d13fb0]

* Merge Update dependencies [caf373d]

* Merge Cron timezone [8e04942]

* Merge Spleling [a37aafc]

* Merge Update README.md [0109a0b,fb15aa4,4e03db7]

* Merge Use the larger of min-OD-instances and (min-OD-percent * current)
[25ac31f]

* gofmt changes

* cronEventAction - fix to logic

need to invert test for needReplaceOnDemandInstances and
onDemandInstance == nil

currently if onDemandInstance == nil methods return and execution stop.
This way the check to terminate a spotInstace if their number is more
than required is never done.

Assume that ASG scale down and terminate the onDemand instances that
autospotting is not terminating [autospotting_min_on_demand_number].
On the next runs onDemandInstance will be nil and spot instances in
excess will not be terminated.

Co-authored-by: Chris <chris.farrenden@domain.com.au>
Co-authored-by: 0x11 <14269809+codenoid@users.noreply.github.com>
Co-authored-by: Jawad <jawad.stouli@gmail.com>
Co-authored-by: Gábor Lipták <gliptak@gmail.com>

.github/FUNDING.yml

@@ -0,0 +1,3 @@
+github: [cristim]
+patreon: cristim
+custom: ["https://www.paypal.me/cristim"]

.github/PULL_REQUEST_TEMPLATE.md

@@ -39,6 +39,8 @@ Gitter and should only be cleared after the PR is merged.
 The code review should be largely a matter of going through this checklist.
 -->
 
+1. [ ] I hereby allow the Copyright holder the rights to distribute this piece of
+   code under any software license.
 1. [ ] The contribution fixes a single existing github issue, and it is linked
    to it.
 1. [ ] The code is as simple as possible, readable and follows the idiomatic Go

.travis.yml

@@ -5,7 +5,7 @@ env:
     # encrypted coveralls secret token, defined as the COVERALLS_TOKEN environment variable
     secure: E0H+b3i171uG8e/w6AVV9HDCCcoReTNlk2+YSENFYmnzxBmvQOo2YwI1jtBRgktG51yZZCzUJZnc0bkaghRT6nS6H8gVUN5GuZjMXdEwY3r+2Zqka6sS1ym+PK6JEA1bRjtX2iQFtk2V68gzUNvjKj8YB2zVJmX6St7tcQ9lBcluQ0xEFu7MZMknYrl9W2NnAhajt27D50uUELtSo3n61TEXXIvFRQyjHCMpGmqeZcXYtiSZAyZjraTsZr9/dngOSw0iExXDDa7cYPHHmmre+3aQUJ9AzotgB+C3+SJ+zOaOGPvVc4PS/G2Wj4siQ1Yv3nl9pvGgxeKtZgVQ92KN9ngdVtRs+XnwMC/x9VJN8vHHgI+coETZ9lhwhjeS8yFIXdBaktXz014ZVrVVpSB+BzoW4r6x/MlmZh0muJNME5OGFXbR0qFaaUB9WdQCst+MhVvSKiZ3Slmck4/3UcyAsuePj0sF1Z/yv9YJR84NTEUdif/PQnBDC1ulhXAIhZ+gm7ZOMZXRW9U7YZlHnFpBg8KoVETlnkT6AjKZ5jy+gz4CgisN2H7OuyjVQm5kIEXPo8ORJqYHPM3tu9HVsCHb6Pm+y9pec/1YXSKtmFbJiYeAmbz3Lp8vVUX5tOo24o0hCyU7qA/ypCUxRoHrHHC+c9u8nQDWGcQBD0azp+dcy2s=
 script:
-- make travisci-docker
+- make ci-docker
 after_success:
 - curl --request POST "https://goreportcard.com/checks" --data "repo=github.com/AutoSpotting/AutoSpotting"
 deploy:

CUSTOM_BUILDS.md

@@ -64,9 +64,12 @@ your CloudFormation or Terraform stack to use those new binaries.
 
 1. Build the code:
 
-   `make build` or `go build` should both work, the only difference is the
-   inclusion of versioning information into the binary, which may not matter
-   much anyway.
+   Run `make build` to generate a binary at `./AutoSpotting`. The resulting
+   binary is suitable for running on AWS Lambda. If you want a binary for
+   running locally and your local environment is not linux/amd64, try running
+   the following:
+
+   `GOOS=$(go env GOOS) GOARCH=$(go env GOARCH) make build` 
 
 ## Running locally ##
 

Dockerfile-build

@@ -1,8 +1,8 @@
-FROM golang:1.13-alpine
+FROM golang:1.14-alpine
 RUN apk add -U --no-cache ca-certificates git make zip
 
 COPY . /src
 WORKDIR /src
 
 # fetch deps and cache them in a Docker storage layer
-RUN go get ./...
+RUN go mod download

LICENSE

@@ -1,18 +1,172 @@
-Copyright (c) 2016-2018 Cristian Măgherușan-Stanciu
-
-Permission is hereby granted, free of charge, to any person obtaining a copy of
-this software and associated documentation files (the "Software"), to deal in
-the Software without restriction, including without limitation the rights to
-use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
-the Software, and to permit persons to whom the Software is furnished to do so,
-subject to the following conditions:
-
-The above copyright notice and this permission notice shall be included in all
-copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
-FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
-COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
-IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
-CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+Open Software License ("OSL") v 3.0
+
+This Open Software License (the "License") applies to any original work of
+authorship (the "Original Work") whose owner (the "Licensor") has placed the
+following licensing notice adjacent to the copyright notice for the Original
+Work:
+
+Licensed under the Open Software License version 3.0
+
+1) Grant of Copyright License. Licensor grants You a worldwide, royalty-free,
+non-exclusive, sublicensable license, for the duration of the copyright, to do
+the following:
+
+  a) to reproduce the Original Work in copies, either alone or as part of a
+  collective work;
+
+  b) to translate, adapt, alter, transform, modify, or arrange the Original
+  Work, thereby creating derivative works ("Derivative Works") based upon the
+  Original Work;
+
+  c) to distribute or communicate copies of the Original Work and Derivative
+  Works to the public, with the proviso that copies of Original Work or
+  Derivative Works that You distribute or communicate shall be licensed under
+  this Open Software License;
+
+  d) to perform the Original Work publicly; and
+
+  e) to display the Original Work publicly.
+
+2) Grant of Patent License. Licensor grants You a worldwide, royalty-free,
+non-exclusive, sublicensable license, under patent claims owned or controlled
+by the Licensor that are embodied in the Original Work as furnished by the
+Licensor, for the duration of the patents, to make, use, sell, offer for sale,
+have made, and import the Original Work and Derivative Works.
+
+3) Grant of Source Code License. The term "Source Code" means the preferred
+form of the Original Work for making modifications to it and all available
+documentation describing how to modify the Original Work. Licensor agrees to
+provide a machine-readable copy of the Source Code of the Original Work along
+with each copy of the Original Work that Licensor distributes. Licensor
+reserves the right to satisfy this obligation by placing a machine-readable
+copy of the Source Code in an information repository reasonably calculated to
+permit inexpensive and convenient access by You for as long as Licensor
+continues to distribute the Original Work.
+
+4) Exclusions From License Grant. Neither the names of Licensor, nor the names
+of any contributors to the Original Work, nor any of their trademarks or
+service marks, may be used to endorse or promote products derived from this
+Original Work without express prior permission of the Licensor. Except as
+expressly stated herein, nothing in this License grants any license to
+Licensor's trademarks, copyrights, patents, trade secrets or any other
+intellectual property. No patent license is granted to make, use, sell, offer
+for sale, have made, or import embodiments of any patent claims other than the
+licensed claims defined in Section 2. No license is granted to the trademarks
+of Licensor even if such marks are included in the Original Work. Nothing in
+this License shall be interpreted to prohibit Licensor from licensing under
+terms different from this License any Original Work that Licensor otherwise
+would have a right to license.
+
+5) External Deployment. The term "External Deployment" means the use,
+distribution, or communication of the Original Work or Derivative Works in any
+way such that the Original Work or Derivative Works may be used by anyone
+other than You, whether those works are distributed or communicated to those
+persons or made available as an application intended for use over a network.
+As an express condition for the grants of license hereunder, You must treat
+any External Deployment by You of the Original Work or a Derivative Work as a
+distribution under section 1(c).
+
+6) Attribution Rights. You must retain, in the Source Code of any Derivative
+Works that You create, all copyright, patent, or trademark notices from the
+Source Code of the Original Work, as well as any notices of licensing and any
+descriptive text identified therein as an "Attribution Notice." You must cause
+the Source Code for any Derivative Works that You create to carry a prominent
+Attribution Notice reasonably calculated to inform recipients that You have
+modified the Original Work.
+
+7) Warranty of Provenance and Disclaimer of Warranty. Licensor warrants that
+the copyright in and to the Original Work and the patent rights granted herein
+by Licensor are owned by the Licensor or are sublicensed to You under the
+terms of this License with the permission of the contributor(s) of those
+copyrights and patent rights. Except as expressly stated in the immediately
+preceding sentence, the Original Work is provided under this License on an "AS
+IS" BASIS and WITHOUT WARRANTY, either express or implied, including, without
+limitation, the warranties of non-infringement, merchantability or fitness for
+a particular purpose. THE ENTIRE RISK AS TO THE QUALITY OF THE ORIGINAL WORK
+IS WITH YOU. This DISCLAIMER OF WARRANTY constitutes an essential part of this
+License. No license to the Original Work is granted by this License except
+under this disclaimer.
+
+8) Limitation of Liability. Under no circumstances and under no legal theory,
+whether in tort (including negligence), contract, or otherwise, shall the
+Licensor be liable to anyone for any indirect, special, incidental, or
+consequential damages of any character arising as a result of this License or
+the use of the Original Work including, without limitation, damages for loss
+of goodwill, work stoppage, computer failure or malfunction, or any and all
+other commercial damages or losses. This limitation of liability shall not
+apply to the extent applicable law prohibits such limitation.
+
+9) Acceptance and Termination. If, at any time, You expressly assented to this
+License, that assent indicates your clear and irrevocable acceptance of this
+License and all of its terms and conditions. If You distribute or communicate
+copies of the Original Work or a Derivative Work, You must make a reasonable
+effort under the circumstances to obtain the express assent of recipients to
+the terms of this License. This License conditions your rights to undertake
+the activities listed in Section 1, including your right to create Derivative
+Works based upon the Original Work, and doing so without honoring these terms
+and conditions is prohibited by copyright law and international treaty.
+Nothing in this License is intended to affect copyright exceptions and
+limitations (including "fair use" or "fair dealing"). This License shall
+terminate immediately and You may no longer exercise any of the rights granted
+to You by this License upon your failure to honor the conditions in Section
+1(c).
+
+10) Termination for Patent Action. This License shall terminate automatically
+and You may no longer exercise any of the rights granted to You by this
+License as of the date You commence an action, including a cross-claim or
+counterclaim, against Licensor or any licensee alleging that the Original Work
+infringes a patent. This termination provision shall not apply for an action
+alleging patent infringement by combinations of the Original Work with other
+software or hardware.
+
+11) Jurisdiction, Venue and Governing Law. Any action or suit relating to this
+License may be brought only in the courts of a jurisdiction wherein the
+Licensor resides or in which Licensor conducts its primary business, and under
+the laws of that jurisdiction excluding its conflict-of-law provisions. The
+application of the United Nations Convention on Contracts for the
+International Sale of Goods is expressly excluded. Any use of the Original
+Work outside the scope of this License or after its termination shall be
+subject to the requirements and penalties of copyright or patent law in the
+appropriate jurisdiction. This section shall survive the termination of this
+License.
+
+12) Attorneys' Fees. In any action to enforce the terms of this License or
+seeking damages relating thereto, the prevailing party shall be entitled to
+recover its costs and expenses, including, without limitation, reasonable
+attorneys' fees and costs incurred in connection with such action, including
+any appeal of such action. This section shall survive the termination of this
+License.
+
+13) Miscellaneous. If any provision of this License is held to be
+unenforceable, such provision shall be reformed only to the extent necessary
+to make it enforceable.
+
+14) Definition of "You" in This License. "You" throughout this License,
+whether in upper or lower case, means an individual or a legal entity
+exercising rights under, and complying with all of the terms of, this License.
+For legal entities, "You" includes any entity that controls, is controlled by,
+or is under common control with you. For purposes of this definition,
+"control" means (i) the power, direct or indirect, to cause the direction or
+management of such entity, whether by contract or otherwise, or (ii) ownership
+of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial
+ownership of such entity.
+
+15) Right to Use. You may use the Original Work in all ways not otherwise
+restricted or conditioned by this License or by law, and Licensor promises not
+to interfere with or be responsible for such uses by You.
+
+16) Modification of This License. This License is Copyright © 2005 Lawrence
+Rosen. Permission is granted to copy, distribute, or communicate this License
+without modification. Nothing in this License permits You to modify this
+License as applied to the Original Work or to Derivative Works. However, You
+may modify the text of this License and copy, distribute or communicate your
+modified version (the "Modified License") and apply it to other original works
+of authorship subject to the following conditions: (i) You may not indicate in
+any way that your Modified License is the "Open Software License" or "OSL" and
+you may not use those names in the name of your Modified License; (ii) You
+must replace the notice specified in the first paragraph above with the notice
+"Licensed under <insert your license name here>" or with a notice of your own
+that is not confusingly similar to the notice in this License; and (iii) You
+may not claim that your original works are open source software unless your
+Modified License has been approved by Open Source Initiative (OSI) and You
+comply with its license review and certification process.