pikesaku’s blog

個人的な勉強メモです。記載内容について一切の責任は持ちません。

awsコマンドのヘルプ

基本"コマンド help"でいけそう。

実行例

$ aws s3 help
S3()                                                                      S3()



NAME
       s3 -

DESCRIPTION
       This  section  explains  prominent concepts and notations in the set of
       high-level S3 commands provided.

   Path Argument Type
       Whenever using a command, at least one path argument must be specified.
       There are two types of path arguments: LocalPath and S3Uri.

       LocalPath: represents the path of a local file or directory.  It can be
       written as an absolute path or relative path.

       S3Uri: represents the location of a S3 object, prefix, or bucket.  This
       must  be  written in the form s3://mybucket/mykey where mybucket is the
       specified S3 bucket, mykey is the specified S3 key.  The path  argument
       must  begin with s3:// in order to denote that the path argument refers
       to a S3 object. Note that prefixes are separated  by  forward  slashes.
       For  example, if the S3 object myobject had the prefix myprefix, the S3
       key would be myprefix/myobject, and if the object  was  in  the  bucket
       mybucket, the S3Uri would be s3://mybucket/myprefix/myobject.

   Order of Path Arguments
       Every  command  takes  one or two positional path arguments.  The first
       path argument represents the source, which is the local  file/directory
       or  S3  object/prefix/bucket  that  is being referenced.  If there is a
       second path argument, it represents the destination, which is the local
       file/directory  or  S3  object/prefix/bucket that is being operated on.
       Commands with only one path argument do not have a destination  because
       the operation is being performed only on the source.

   Single Local File and S3 Object Operations
       Some  commands  perform operations only on single files and S3 objects.
       The following commands are single file/object operations if no --recur-
       sive flag is provided.

          o cp

          o mv

          o rm

       For  this  type of operation, the first path argument, the source, must
       exist and be a local file or S3 object.  The second path argument,  the
       destination,  can  be  the  name  of  a local file, local directory, S3
       object, S3 prefix, or S3 bucket.

       The destination is indicated as a local directory,  S3  prefix,  or  S3
       bucket if it ends with a forward slash or back slash.  The use of slash
       depends on the path argument type.  If the path argument  is  a  Local-
       Path,  the type of slash is the separator used by the operating system.
       If the path is a S3Uri, the forward slash must always be  used.   If  a
       slash  is at the end of the destination, the destination file or object
       will adopt the name of the source file or object.  Otherwise, if  there
       is no slash at the end, the file or object will be saved under the name
       provided.  See examples in cp and mv to illustrate this description.

   Directory and S3 Prefix Operations
       Some commands only perform operations on the contents of a local direc-
       tory  or  S3 prefix/bucket.  Adding or omitting a forward slash or back
       slash to the end of any path argument, depending on its type, does  not
       affect  the  results  of  the  operation.   The following commands will
       always result in a directory or S3 prefix/bucket operation:

       o sync

       o mb

       o rb

       o ls

   Use of Exclude and Include Filters
       Currently, there is no support for the use of UNIX style wildcards in a
       command's  path  arguments.   However,  most  commands  have  --exclude
       "<value>" and --include  "<value>"  parameters  that  can  achieve  the
       desired  result.   These  parameters perform pattern matching to either
       exclude or include a particular file or object.  The following  pattern
       symbols are supported.

          o *: Matches everything

          o ?: Matches any single character

          o [sequence]: Matches any character in sequence

          o [!sequence]: Matches any character not in sequence

       Any  number of these parameters can be passed to a command.  You can do
       this by providing an --exclude or --include  argument  multiple  times,
       e.g.   --include  "*.txt"  --include  "*.png".  When there are multiple
       filters, the rule is the filters that appear later in the command  take
       precedence  over filters that appear earlier in the command.  For exam-
       ple, if the filter parameters passed to the command were

          --exclude "*" --include "*.txt"

       All files will be excluded from the command  except  for  files  ending
       with  .txt   However, if the order of the filter parameters was changed
       to

          --include "*.txt" --exclude "*"

       All files will be excluded from the command.

       Each filter is evaluated against the source directory.  If  the  source
       location is a file instead of a directory, the directory containing the
       file is used as the source directory.  For example, suppose you had the
       following directory structure:

          /tmp/foo/
            .git/
            |---config
            |---description
            foo.txt
            bar.txt
            baz.jpg

       In  the  command aws s3 sync /tmp/foo s3://bucket/ the source directory
       is /tmp/foo.  Any include/exclude filters will be  evaluated  with  the
       source  directory prepended.  Below are several examples to demonstrate
       this.

       Given the directory structure above and the command aws s3 cp  /tmp/foo
       s3://bucket/  --recursive --exclude ".git/*", the files .git/config and
       .git/description will be excluded from the files to upload because  the
       exclude  filter  .git/*  will  have the source prepended to the filter.
       This means that:

          /tmp/foo/.git/* -> /tmp/foo/.git/config       (matches, should exclude)
          /tmp/foo/.git/* -> /tmp/foo/.git/description  (matches, should exclude)
          /tmp/foo/.git/* -> /tmp/foo/foo.txt  (does not match, should include)
          /tmp/foo/.git/* -> /tmp/foo/bar.txt  (does not match, should include)
          /tmp/foo/.git/* -> /tmp/foo/baz.jpg  (does not match, should include)

       The command aws s3  cp  /tmp/foo/  s3://bucket/  --recursive  --exclude
       "ba*" will exclude /tmp/foo/bar.txt and /tmp/foo/baz.jpg:

          /tmp/foo/ba* -> /tmp/foo/.git/config      (does not match, should include)
          /tmp/foo/ba* -> /tmp/foo/.git/description (does not match, should include)
          /tmp/foo/ba* -> /tmp/foo/foo.txt          (does not match, should include)
          /tmp/foo/ba* -> /tmp/foo/bar.txt  (matches, should exclude)
          /tmp/foo/ba* -> /tmp/foo/baz.jpg  (matches, should exclude)

       Note that, by default, all files are included.  This means that provid-
       ing only an --include filter will not  change  what  files  are  trans-
       ferred.   --include  will only re-include files that have been excluded
       from an --exclude filter.  If you only want to upload files with a par-
       ticular extension, you need to first exclude all files, then re-include
       the files with the particular extension.  This command will upload only
       files ending with .jpg:

          aws s3 cp /tmp/foo/ s3://bucket/ --recursive --exclude "*" --include "*.jpg"

       If  you wanted to include both .jpg files as well as .txt files you can
       run:

          aws s3 cp /tmp/foo/ s3://bucket/ --recursive \
              --exclude "*" --include "*.jpg" --include "*.txt"

SYNOPSIS
          aws s3 <Command> [<Arg> ...]

OPTIONS
       None

AVAILABLE COMMANDS
       o cp

       o ls

       o mb

       o mv

       o presign

       o rb

       o rm

       o sync

       o website


サブコマンドwebsiteのヘルプ確認

$ aws s3 website help
WEBSITE()                                                            WEBSITE()



NAME
       website -

DESCRIPTION
       Set the website configuration for a bucket.

SYNOPSIS
            website
          <S3Uri>
          [--index-document <value>]
          [--error-document <value>]

OPTIONS
       paths (string)

       --index-document  (string)  A suffix that is appended to a request that
       is for a directory on the website  endpoint  (e.g.  if  the  suffix  is
       index.html and you make a request to samplebucket/images/ the data that
       is returned will be for the object with the key name images/index.html)
       The suffix must not be empty and must not include a slash character.

       --error-document  (string)  The object key name to use when a 4XX class
       error occurs.

EXAMPLES
       The following command configures a bucket named my-bucket as  a  static
       website:

          aws s3 website s3://my-bucket/ --index-document index.html --error-document error.html

       The index document option specifies the file in my-bucket that visitors
       will be directed to when they navigate to  the  website  URL.  In  this
       case,  the  bucket is in the us-west-2 region, so the site would appear
       at http://my-bucket.s3-website-us-west-2.amazonaws.com.

       All files in the bucket that appear on the static site must be  config-
       ured  to  allow  visitors to open them. File permissions are configured
       separately from the bucket website configuration.  For  information  on
       hosting  a static website in Amazon S3, see Hosting a Static Website in
       the Amazon Simple Storage Service Developer Guide.



                                                                     WEBSITE()

ログ画像化(線表示)

コード

apache_log_trans_to_image.py

# -*- coding:utf-8 -*-
import argparse
import apache_log_trans_to_image_lib as alti

parser = argparse.ArgumentParser(description='apache log to graph')
parser.add_argument('log', help='log file', type=argparse.FileType('r'))
parser.add_argument('--hash', help='define hash type', type=str, choices=['md5', 'sha256'], default='md5')
parser.add_argument('--unit', help='unit of urls', type=int, default=2)
args = parser.parse_args()


if __name__ == '__main__':
    data = alti.get_data(args.log, args.unit)
    data = alti.change_data_for_graph(data, args.hash)
    alti.output_graph(data, args.unit, args.hash)

apache_log_trans_to_image_lib.py

# -*- coding:utf-8 -*-


def get_data(log, unit):
    import apache_log_parser
    import itertools

    def chk_key(line):
        required_key = ('request_url_path', 'remote_host')
        for key in required_key:
            if not key in line:
                return False
        return True

    def chk_ext(line):
        request_url_path = line['request_url_path']
        except_ext = ('gif', 'jpg', 'png', 'ico', 'css', 'js', 'woff', 'ttf', 'svg')
        ext = request_url_path.split('.')[-1].lower()
        if ext in except_ext:
            return False
        return True

    data = dict()
    parser = apache_log_parser.make_parser('%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"')

    for line in log:
        line = line.strip()
        line = parser(line)
        if not chk_key(line):
            continue
        if not chk_ext(line):
            continue
        host = line['remote_host']
        request_url_path = line['request_url_path']
        if host in data:
            data[host].append(request_url_path)
        else:
            data[host] = [request_url_path]

    for host,request_url_path_list in data.items():
        request_url_path_list = list(itertools.zip_longest(*[iter(request_url_path_list)]*unit))
        request_url_path_list[-1] = [request_url_path for request_url_path in request_url_path_list[-1] if request_url_path is not None]
        data[host] = request_url_path_list

    return data


def change_data_for_graph(data, h):
    changed_data = dict()
    for host,request_url_path_list in data.items():
        units_of_nums = list()
        for part in request_url_path_list:
            units_of_nums.append([trans_str_to_num(url, h) for url in part])
        changed_data[host] = units_of_nums
    return changed_data


def trans_str_to_num(s, h):
    import hashlib
    import re
    s = s.encode('UTF-8')
    if h == 'md5':
        m = hashlib.md5()
    if h == 'sha256':
        m = hashlib.sha256()
    m.update(s)
    h = m.hexdigest()
    # hは16進数32桁
    # 4桁づつ、リストにする。
    # https://stackoverflow.com/questions/13673060/split-string-into-strings-by-length
    nums = [ int(i, 16) for i in re.split('(.{4})', h)[1::2] ]
    return nums


def output_graph(data, unit, h):
    import numpy as np
    import matplotlib.pyplot as plt

    if h == 'md5':
        xtick = 8
    if h == 'sha256':
        xtick = 16

    # https://stackoverflow.com/questions/24943991/change-grid-interval-and-specify-tick-labels-in-matplotlib
    for ip,units_of_nums in data.items():
        seq = 0
        for unit_of_nums in units_of_nums:
            nums = list()
            for num in unit_of_nums:
                nums.extend(num)
            x, y = (range(len(nums)), nums)
            fig, ax = plt.subplots()
            major_xticks = np.arange(0, unit*xtick+1, xtick)
            minor_xticks = np.arange(0, unit*xtick+1, 1)
            major_yticks = np.arange(0, 65535+1, 10000)
            minor_yticks = np.arange(0, 65535+1, 1000)

            ax.set_xticks(major_xticks)
            ax.set_xticks(minor_xticks, minor=True)
            ax.set_yticks(major_yticks)
            ax.set_yticks(minor_yticks, minor=True)
            ax.grid(which='both')
            ax.grid(which='minor', alpha=0.2)
            ax.grid(which='major', alpha=0.8)
            plt.plot(x, y, color='red', lw=0.5)

            # 目盛を表示する場合、以下をコメントアウト
            plt.yticks(color='None')
            plt.xticks(color='None')
            #

            plt.xlim([0, unit*xtick])
            plt.ylim([0, 65535])
            plt.savefig(ip + '_' + str(seq) + '.png')
            plt.close()
            seq += 1

アウトプット

以下ログの場合(--unitはデフォルト10)

192.168.56.1 - - [08/Jul/2018:12:24:05 +0900] "GET /wp/wp-admin/install.php HTTP/1.1" 500 3606 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.162 Safari/537.36"
192.168.56.1 - - [08/Jul/2018:12:24:05 +0900] "GET /wp/wp-admin/install.php HTTP/1.1" 500 3606 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.162 Safari/537.36"
192.168.56.1 - - [08/Jul/2018:12:24:05 +0900] "GET /wp/wp-admin/install.php HTTP/1.1" 500 3606 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.162 Safari/537.36"
192.168.56.1 - - [08/Jul/2018:12:34:54 +0900] "GET /wp/wp-admin/update-core.php HTTP/1.1" 200 16579 "http://192.168.56.101/wp/wp-admin/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.162 Safari/537.36"
192.168.56.1 - - [08/Jul/2018:12:34:54 +0900] "GET /wp/wp-admin/update-core.php HTTP/1.1" 200 16579 "http://192.168.56.101/wp/wp-admin/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.162 Safari/537.36"
192.168.56.1 - - [08/Jul/2018:12:34:54 +0900] "GET /wp/wp-admin/update-core.php HTTP/1.1" 200 16579 "http://192.168.56.101/wp/wp-admin/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.162 Safari/537.36"

192.168.56.1_0.png
f:id:pikesaku:20181125013238p:plain
 

以下ログの場合(--unitはデフォルト10)

192.168.56.1 - - [08/Jul/2018:12:24:05 +0900] "GET /wp/wp-admin/install.php HTTP/1.1" 500 3606 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.162 Safari/537.36"
192.168.56.1 - - [08/Jul/2018:12:24:05 +0900] "GET /wp/wp-admin/install.php HTTP/1.1" 500 3606 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.162 Safari/537.36"
192.168.56.1 - - [08/Jul/2018:12:24:05 +0900] "GET /wp/wp-admin/install.php HTTP/1.1" 500 3606 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.162 Safari/537.36"
192.168.56.1 - - [08/Jul/2018:12:24:05 +0900] "GET /wp/wp-admin/install.php HTTP/1.1" 500 3606 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.162 Safari/537.36"
192.168.56.1 - - [08/Jul/2018:12:24:05 +0900] "GET /wp/wp-admin/install.php HTTP/1.1" 500 3606 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.162 Safari/537.36"
192.168.56.1 - - [08/Jul/2018:12:24:05 +0900] "GET /wp/wp-admin/install.php HTTP/1.1" 500 3606 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.162 Safari/537.36"
192.168.56.1 - - [08/Jul/2018:12:24:05 +0900] "GET /wp/wp-admin/install.php HTTP/1.1" 500 3606 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.162 Safari/537.36"
192.168.56.1 - - [08/Jul/2018:12:24:05 +0900] "GET /wp/wp-admin/install.php HTTP/1.1" 500 3606 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.162 Safari/537.36"
192.168.56.1 - - [08/Jul/2018:12:24:05 +0900] "GET /wp/wp-admin/install.php HTTP/1.1" 500 3606 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.162 Safari/537.36"
192.168.56.1 - - [08/Jul/2018:12:24:05 +0900] "GET /wp/wp-admin/install.php HTTP/1.1" 500 3606 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.162 Safari/537.36"
192.168.56.1 - - [08/Jul/2018:12:24:05 +0900] "GET /wp/wp-admin/install.php HTTP/1.1" 500 3606 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.162 Safari/537.36"

192.168.56.1_0.png
f:id:pikesaku:20181125013028p:plain
192.168.56.1_1.png
f:id:pikesaku:20181125013043p:plain

AWS IAM調査メモ

メモ

・アカウント作成時のID/PASSがルートユーザー
・ルートユーザーはユーザー作成以外では基本使わないのがベストプラクティクス。
・ユーザー、グループを作成し、グループに利用可能なサービスを設定

ユーザー作成画面

ユーザー名とアクセス方法を指定
f:id:pikesaku:20181021100702p:plain
 
グループの作成をクリックし、所属するグループを作成
f:id:pikesaku:20181021101058p:plain
 
ポリシーで利用を許可するサービスを指定(今回はS3指定)
f:id:pikesaku:20181021101305p:plain
 

動作確認

ユーザー作成後、以下情報が得られる。

項目 説明
アクセスキーID SDK/CLIからのアクセス時に必要な情報
シークレットアクセスキー SDK/CLIからのアクセス時に必要な情報
コンソールアクセスリンク 作成したユーザー用ログインURL

 
コンソールアクセスリンクにアクセスしログイン。許可されてないEC2インスタンス起動操作をするエラーになる。
f:id:pikesaku:20181021103508p:plain
 

ロール

・アクセスキー、シークレットが漏洩し、莫大な請求されるケースあり。
プログラムではアクセスキー/シークレットキーを使わずにRoleを利用する | DevelopersIO
・アクセスキーなどを書いたPrgをgitにアップロードしてしまい発生する
・アクセスキーは使わずにプログラムからAWSサービスを利用するにはロールを使う
・ロールはEC2の起動時に割り当てる。該当EC2からアクセスキーを使わずにプログラムアクセスが可能

AWS Cloudwatch調査メモ

メモ

AWSリソースとアプリケーションのリアルタイムモニタリング機能
・リソースとアプリケーションに設定された変数情報を参照してモニタリングする
・ルールを定義しマッチした場合、自動的にリソース変更が可能(使いすぎたら止める等、オートスケールでも利用)
・以下でアクセス可能
 Amazon CloudWatch コンソール
  https://console.aws.amazon.com/cloudwatch/
 AWS CLI
 CloudWatch API
 AWS SDK
・以下サービスと併用される

サービス 説明
Amazon Simple Notification Service (Amazon SNS) エンドポイント・クライアントへのメッセージ送信
Amazon EC2 Auto Scaling オートスケール
AWS CloudTrail Amazon CloudWatch API 宛ての呼び出しモニタリング
AWS Identity and Access Management (IAM) アクセスコントロール