EzDev.org

boto

For the latest version of boto, see <a href="https://github.com/boto/boto3">https://github.com/boto/boto3</a> -- Python interface to Amazon Web Services boto: A Python interface to Amazon Web Services &mdash; boto v2.38.0


How to mix Django, Uploadify, and S3Boto Storage Backend?

Background

I'm doing fairly big file uploads on Django. File size is generally 10MB-100MB.

I'm on Heroku and I've been hitting the request timeout of 30 seconds.

The Beginning

In order to get around the limit, Heroku's recommendation is to upload from the browser DIRECTLY to S3.

Amazon documents this by showing you how to write an HTML form to perform the upload.

Since I'm on Django, rather than write the HTML by hand, I'm using django-uploadify-s3 (example). This provides me with an SWF object, wrapped in JS, that performs the actual upload.

This part is working fine! Hooray!

The Problem

The problem is in tying that data back to my Django model in a sane way. Right now the data comes back as a simple URL string, pointing to the file's location.

However, I was previously using S3 Boto from django-storages to manage all of my files as FileFields, backed by the delightful S3BotoStorageFile.

To reiterate, S3 Boto is working great in isolation, Uploadify is working great in isolation, the problem is in putting the two together.

My understanding is that the only way to populate the FileField is by providing both the filename AND the file content. When you're uploading files from the browser to Django, this is no problem, as Django has the file content in a buffer and can do whatever it likes with it. However, when doing direct-to-S3 uploads like me, Django only receives the file name and URL, not the binary data, so I can't properly populate the FieldFile.

Cry For Help

Anyone know a graceful way to use S3Boto's FileField in conjunction with direct-to-S3 uploading?

Else, what's the best way to manage an S3 file just based on its URL? Including setting expiration, key id, etc.

Many thanks!


Source: (StackOverflow)

Django + S3 (boto) + Sorl Thumbnail: Suggestions for optimisation

I am using S3 storage backend across a Django site I am developing, both to reduce load from the EC2 server(s), and to allow multiple webservers (redundancy, load balancing) access the same set of uploaded media.

Sorl.thumbnail (v11) template tags are being used in our templates to allow flexible image resizing/cropping.

Performance on media-rich pages is not very good, and when a page containing thumbnails needing to be generated for the first time is accessed, the requests even time out.

I understand that this is due to sorl thumbnail checking/downloading the original image from S3 (which could be quite large and high resolution), and rendering/checking/uploading the thumbnail.

What would you suggest is the best solution to this setup?

I have seen suggestions of storing a local copy of files in addition to the S3 copy (not to great when a couple of server are being used for load balancing). Also I've seen it suggested to store 0-byte files to fool sorl.thumbnail.

Are there any other suggestions or better ways of approaching this?


Source: (StackOverflow)

Downloading the files from s3 recursively using boto python.

I have a bucket in s3, which has deep directory structure. I wish I could download them all at once. My files look like this :

foo/bar/1. . 
foo/bar/100 . . 

Are there any ways to download these files recursively from the s3 bucket using boto lib in python?

Thanks in advance.


Source: (StackOverflow)

How do I test a module that depends on boto and an Amazon AWS service?

I'm writing a very small Python ORM around boto.dynamodb.layer2. I would like to write tests for it, but I don't want the tests to actually communicate with AWS, as this would require complicated setup, credentials, network access, etc.

Since I plan to open source the module, including credentials in the source seems like a bad idea since I will get charged for usage, and including credentials in the environment is a pain.

Coupling my tests to the network seems like a bad idea, as it makes the tests run slower, or may cause tests to fail due to network errors or throttling. My goal is not to test boto's DynamoDB interface, or AWS. I just want to test my own code.

I plan to use unittest2 to write the tests and mock to mock out the parts of boto that hit the network, but I've never done this before, so my question boils down to these:

  1. Am I going about this the right way?
  2. Has anyone else done this?
  3. Are there any particular points in the boto.dynamodb interface that would be best to mock out?

Source: (StackOverflow)

How to auto assign public ip to EC2 instance with boto

I have to start a new machine with ec2.run_instances in a given subnet but also to have a public ip auto assigned (not fixed elastic ip).

When one starts a new machine from the Amazon's web EC2 Manager via the Request Instance (Instance details) there is a check-box called Assign Public IP to Auto-assign Public IP. See it highlighted in the screenshot:

Request Instance wizard

How can I achieve that check-box functionality with boto?


Source: (StackOverflow)

How to launch EC2 instance with Boto, specifying size of EBS?

I'm using boto/python to launch a new EC2 instance that boots from an EBS volume. At the time I launch the instance, I'd like to override the default size of the booting EBS volume.

I found no boto methods or parameters that might fit into my launch code:

ec2 = boto.connect_ec2( ACCESS_KEY, SECRET_KEY, region=region )

reservation = ec2.run_instances( image_id=AMI_ID, 
                                 key_name=EC2_KEY_HANDLE, 
                                 instance_type=INSTANCE_TYPE,
                                 security_groups = [ SECGROUP_HANDLE, ] )

The web page below shows how to increase the size of a running EC2-instance's EBS volume using command-line tools, but I'd like to use boto at the time the EC2 instance is specified:

http://alestic.com/2010/02/ec2-resize-running-ebs-root

Thank you.


Source: (StackOverflow)

How do I install boto?

So that I am able to work with it within my python scripts?


Source: (StackOverflow)

boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden

I'm trying to get django to upload static files to S3, but istead I'm getting a 403 forbidden error, and I'm not sure why.

Full Stacktrace:

Traceback (most recent call last):
  File "manage.py", line 14, in <module>
    execute_manager(settings)
  File "/home/levi/Projects/DoneBox/.virtualenv/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 438, in execute_manager
    utility.execute()
  File "/home/levi/Projects/DoneBox/.virtualenv/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 379, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/levi/Projects/DoneBox/.virtualenv/local/lib/python2.7/site-packages/django/core/management/base.py", line 191, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/home/levi/Projects/DoneBox/.virtualenv/local/lib/python2.7/site-packages/django/core/management/base.py", line 220, in execute
    output = self.handle(*args, **options)
  File "/home/levi/Projects/DoneBox/.virtualenv/local/lib/python2.7/site-packages/django/core/management/base.py", line 351, in handle
    return self.handle_noargs(**options)
  File "/home/levi/Projects/DoneBox/.virtualenv/local/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 89, in handle_noargs
    self.copy_file(path, prefixed_path, storage, **options)
  File "/home/levi/Projects/DoneBox/.virtualenv/local/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 184, in copy_file
    if not self.delete_file(path, prefixed_path, source_storage, **options):
  File "/home/levi/Projects/DoneBox/.virtualenv/local/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 115, in delete_file
    if self.storage.exists(prefixed_path):
  File "/home/levi/Projects/DoneBox/.virtualenv/local/lib/python2.7/site-packages/storages/backends/s3boto.py", line 209, in exists
    return k.exists()
  File "/home/levi/Projects/DoneBox/.virtualenv/local/lib/python2.7/site-packages/boto/s3/key.py", line 391, in exists
    return bool(self.bucket.lookup(self.name))
  File "/home/levi/Projects/DoneBox/.virtualenv/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 143, in lookup
    return self.get_key(key_name, headers=headers)
  File "/home/levi/Projects/DoneBox/.virtualenv/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 208, in get_key
    response.status, response.reason, '')
boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden

Contents of settings.py:

import os
DIRNAME = os.path.dirname(__file__)
# Django settings for DoneBox project.

DEBUG = True
TEMPLATE_DEBUG = DEBUG

ADMINS = (
    # ('Your Name', 'your_email@example.com'),
)

MANAGERS = ADMINS

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3', # Add 'postgresql_psycopg2', 'postgresql', 'mysql', 'sqlite3' or 'oracle'.
        'NAME': os.path.join(DIRNAME, "box.sqlite"),                      # Or path to database file if using sqlite3.
        'USER': '',                      # Not used with sqlite3.
        'PASSWORD': '',                  # Not used with sqlite3.
        'HOST': '',                      # Set to empty string for localhost. Not used with sqlite3.
        'PORT': '',                      # Set to empty string for default. Not used with sqlite3.
    }
}

# Local time zone for this installation. Choices can be found here:
# http://en.wikipedia.org/wiki/List_of_tz_zones_by_name
# although not all choices may be available on all operating systems.
# On Unix systems, a value of None will cause Django to use the same
# timezone as the operating system.
# If running in a Windows environment this must be set to the same as your
# system time zone.
TIME_ZONE = 'America/Denver'

# Language code for this installation. All choices can be found here:
# http://www.i18nguy.com/unicode/language-identifiers.html
LANGUAGE_CODE = 'en-us'

SITE_ID = 1

# If you set this to False, Django will make some optimizations so as not
# to load the internationalization machinery.
USE_I18N = True

# If you set this to False, Django will not format dates, numbers and
# calendars according to the current locale
USE_L10N = True

# Absolute filesystem path to the directory that will hold user-uploaded files.
# Example: "/home/media/media.lawrence.com/media/"
MEDIA_ROOT = ''

# URL that handles the media served from MEDIA_ROOT. Make sure to use a
# trailing slash.
# Examples: "http://media.lawrence.com/media/", "http://example.com/media/"
MEDIA_URL = "d1eyn4cjl5vzx0.cloudfront.net"

# Absolute path to the directory static files should be collected to.
# Don't put anything in this directory yourself; store your static files
# in apps' "static/" subdirectories and in STATICFILES_DIRS.
# Example: "/home/media/media.lawrence.com/static/"
STATIC_ROOT = os.path.join(DIRNAME, "static")

# URL prefix for static files.
# Example: "http://media.lawrence.com/static/"
STATIC_URL = "d280kzug7l5rug.cloudfront.net"

# URL prefix for admin static files -- CSS, JavaScript and images.
# Make sure to use a trailing slash.
# Examples: "http://foo.com/static/admin/", "/static/admin/".
ADMIN_MEDIA_PREFIX = '/static/admin/'

# Additional locations of static files
STATICFILES_DIRS = (
    # Put strings here, like "/home/html/static" or "C:/www/django/static".
    # Always use forward slashes, even on Windows.
    # Don't forget to use absolute paths, not relative paths.
    os.path.join(DIRNAME, "main", "static"),
)

# List of finder classes that know how to find static files in
# various locations.
STATICFILES_FINDERS = (
    'django.contrib.staticfiles.finders.FileSystemFinder',
    'django.contrib.staticfiles.finders.AppDirectoriesFinder',
    'django.contrib.staticfiles.finders.DefaultStorageFinder',
)

# Make this unique, and don't share it with anybody.
SECRET_KEY = '<snip>'

# List of callables that know how to import templates from various sources.
TEMPLATE_LOADERS = (
    'django.template.loaders.filesystem.Loader',
    'django.template.loaders.app_directories.Loader',
    'django.template.loaders.eggs.Loader',
)

MIDDLEWARE_CLASSES = (
    'django.middleware.common.CommonMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
)

ROOT_URLCONF = 'DoneBox.urls'

TEMPLATE_DIRS = (
    # Put strings here, like "/home/html/django_templates" or "C:/www/django/templates".
    # Always use forward slashes, even on Windows.
    # Don't forget to use absolute paths, not relative paths.
    os.path.join(DIRNAME, "main", "templates"),
    os.path.join(DIRNAME, "templates"),
    os.path.join(DIRNAME, "basic", "blog", "templates"),
)

INSTALLED_APPS = (
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.sites',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'django.contrib.sitemaps',
    # Uncomment the next line to enable the admin:
    'django.contrib.admin',
    # Uncomment the next line to enable admin documentation:
    'storages',
    'django.contrib.admindocs',
    'main',
    'contacts',
    'piston',
    'registration',
#    'contact_form',
    'basic',
    'basic.blog',
)

# A sample logging configuration. The only tangible logging
# performed by this configuration is to send an email to
# the site admins on every HTTP 500 error.
# See http://docs.djangoproject.com/en/dev/topics/logging for
# more details on how to customize your logging configuration.
LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'handlers': {
        'mail_admins': {
            'level': 'ERROR',
            'class': 'django.utils.log.AdminEmailHandler'
        }
    },
    'loggers': {
        'django.request': {
            'handlers': ['mail_admins'],
            'level': 'DEBUG',
            'propagate': True,
        },
        'django.db.backends': {
            'handlers': ['mail_admins'],
            'level': 'DEBUG',
            'propagate': True,
        }
    }
}

DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
AWS_ACCESS_KEY_ID = '<snip>'
AWS_SECRET_ACCESS_KEY = '<snip>'
STATICFILES_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
AWS_STORAGE_BUCKET_NAME = "donebox-static"
STATIC_FILES_BUCKET = "donebox-static"
MEDIA_FILES_BUCKET = "donebox-media"
ACCOUNT_ACTIVATION_DAYS = 7

EMAIL_HOST = "email-smtp.us-east-1.amazonaws.com"
EMAIL_HOST_USER = '<snip>'
EMAIL_HOST_PASSWORD = '<snip>'
EMAIL_PORT = 587
EMAIL_USE_TLS = True
TEMPLATE_CONTEXT_PROCESSORS = (
    "django.contrib.auth.context_processors.auth",
     "django.core.context_processors.debug",
     "django.core.context_processors.i18n",
     "django.core.context_processors.media",
     "django.core.context_processors.static",
     "django.contrib.messages.context_processors.messages",
     "DoneBox.main.context_processors_PandC",
     )

Contents of requirements.pip:

django==1.3
django-storages==1.1.4
django-registration==0.8
django-piston==0.2.3
django-tagging==0.3.1
django-extensions==0.8
BeautifulSoup==3.2.1
boto==2.4.1
mysql-python==1.2.3
tweepy==1.9
feedparser==5.1.2
pycrypto==2.6

A google search for this exception doesn't turn up anything interesting. I suspect I mis-configured things, although I'm not sure. Could someone point me in the right direction? Thank you for your time and consideration.


Source: (StackOverflow)

Django Storage Backend for S3

I'm looking for a good Django custom storage backend for use with Amazon S3.

I've been googling around and found a lot of blog posts with code snippets or half-baked gist.github.com one-off jobs. But I can't seem to find a solid, well-tested one.

Is there a widely accepted standard Amazon S3 Django custom storage backend out there? It doesn't particularly matter to me what Python backend library it uses--i.e., either S3.py or boto are fine.


Source: (StackOverflow)

amazon S3 boto - how to create folder?

how can i create a folder under a bucket using boto library for amazon s3, i followed the manual, and create keys with contents with permission, metadata etc, but no where in the boto's documentation say how to create folders under bucket, or create folder under folders in bucket.


Source: (StackOverflow)

Amazon S3 boto: How do you rename a file in a bucket?

How do you rename a S3 key in a bucket with boto?


Source: (StackOverflow)

Disable boto logging without modifying the boto files

I am using the Boto library to talk to AWS. I want to disable logging. (Or redirect to /dev/null or other file). I cant find an obvious way to do this. I tried this, but that doesn't seem to help.

import boto
boto.set_file_logger('boto', 'logs/boto.log')

This says it is possible, http://developer.amazonwebservices.com/connect/thread.jspa?messageID=52727&#52727 but AFAIK the documentation doesnt tell how.


Source: (StackOverflow)

Signature/Expires/Access Key ID appearing in URL Params. Django/Boto/S3

Thumbnail URLS have params in them. I'm not sure why. I think they related to sorl-thumbnail or boto storage we're using. How do I remove the additional params from the URL? Are they coming from boto/s3 or sorl. I've no idea how to debug.


Source: (StackOverflow)

Boto EC2: Create an instance with tags

Is there a way with the boto python API to specify tags when creating an instance? I'm trying to avoid having to create an instance, fetch it and then add tags. It would be much easier to have the instance either pre-configured to have certain tags or to specify tags when I execute the following command:

ec2server.create_instance(
        ec2_conn, ami_name, security_group, instance_type_name, key_pair_name, user_data
    )

Source: (StackOverflow)

How to change metadata on an object in Amazon S3

If you have already uploaded an object to an Amazon S3 bucket, how do you change the metadata using the API? It is possible to do this in the AWS Management Console, but it is not clear how it could be done programmatically. Specifically, I'm using the boto API in Python and from reading the source it is clear that using key.set_metadata only works before the object is created as it just effects a local dictionary.


Source: (StackOverflow)