Parallel Batch Request Handling

 

Picture this:

You find yourself with a big list of resource identifiers

You want to make get information for each id via a http request

You want to batch the resource identifiers rather than making individual http calls

You want to make multiple parallel requests for these batches

You don’t want to manage multiple thread access to a single response collection

 

You have come to the right place!

 

There are multiple approaches I can think of, but here’s one that may fit your use case:

 

Get Method:

The code in the method itself is not that important!
what you should take from it is that it’s an ordinary async/await method that takes a list of Ids and returns the results for same.

 

To Parallelizm and beyond

 

Let’s unravel what’s happening above.

 

Firstly we are using .net6 where we have the Chunk method from the framework (gone are the days of writing this by hand thankfully!),
the chunk method basically takes the ids list and breaks it into a list of smaller lists of size ‘batchSize’ or smaller.

e.g.

If we chunked [1,2,3,4,5,6,7,8,9,10,11] with a batch size of 2, then we’d end up with

[[1,2],[3,4],[5,6],[7,8],[9,10],[11]]

 

 

Secondly we pass these smaller arrays to the GetIds call, by using a Linq Select expression.

We await the result of all these Selected tasks via Task.WhenAll

Lastly to combine all the individual batched responses we use the Linq SelectMany expression.

 

I like this approach as it is terse, concise and propagates exceptions without wrapping them in Aggregates.

Setting your C# language level

The C# compiler defaults to the latest major version of the language that has been released. You may choose to compile any project using a new point release of the language. Choosing a newer version of the language enables your project to make use of the latest language features. In other scenarios, you may need to validate that a project compiles cleanly when using an older version of the language.

This capability decouples the decision to install new versions of the SDK and tools in your development environment from the decision to incorporate new language features in a project. You can install the latest SDK and tools on your build machine. Each project can be configured to use a specific version of the language for its build

Screenshot shows me selecting C# 7.2 for a .net core 2.1 application by changing the advanced options of the project properties build pane

C# Concurrency lesson–SemaphoreSlim

In windows we have two types of semaphores, local and named system semaphores.

You can think of a semaphore as a bounder in a nightclub, the responsibility is to only allow a number of people into the club at any one time.

.net has a lightweight semaphore, ‘SemaphoreSlim’ that can be used for local communication (that is to say, system synchronization is not supported)

If you run the code above (e.g. in .net core 2.1 project) you will be presented with the following result

What is happening is that all the tasks try get access to the semaphore, they are all initially blocked until

semaphore.Release(3);

is called which allows 3 tasks to enter at any one time.

Out with the old in with the new(er)

With 2016 drawing to a close and 2017 already in full swing for me, I thought this was a good opportunity to reflect on how 2016 went and what 2017 has in store for me from a technological point of view.

2016

If asked how 2016 was from a professional perspective I’d probably try to sum it up as follows “Technology continued to roll out at an ever increasing pace, not only was new technology appearing faster than ever before, existing technology stacks started iterate and churn under our feet!”

Nearing the end of 2016 was where I finally admitted defeat and realized that I can’t keep up with everything and I while I sure am greedy and to know everything about everything, it was getting to the point that I was becoming a ‘Jack of all trades and master of none’ dare I say a full stack developer! I’d actually like to think I’m master of some, but certainly it was a big effort to stay on top of everything.

What did I get up to?

Azure : I got certified in Azure this was without doubt my most prized professional achievement of 2016, I’ve been using Azure for years and I feel quite confident in acclaiming it to be the best public could in the world today.
I’ve also started work on a state of the art data distribution network using Server-less architecture. I finally got down and dirty with Swagger/API Apps/LogicApps/AzureFunctions.
I got a lot better at networking, Load balancing resiliency, Azure/AWS causes a, devops inner persona to ooze its way to the top.
I listened with baited breath to the Azure Weekly Azure Podcast to see what was new (and always scratched my head when Cale got excited about BlockChains, perhaps next year I’ll look back and kick my self for not being an early adopter, it does seem to be an area that’s heating up).

AWS: I got certified as an AWS Solutions Architect, it was great to get a better understanding of AWS and indeed for a few offerings they I’d choose them over Azure. Got heavily involved in AWS CloudFormation and helped regain some control on AWS madness.

Google Cloud: I spent a few weeks playing with it just to see how it’s coming along, at least now I’m somewhat informed but I’d only consider myself as a beginner (I’d consider GoogleCloud a beginner also , unless they put in a massive investment into the portal and services, they simply can’t compete with Azure and AWS.

Docker: I can create images, start stop then understand volumes, I didn’t get as far as any of the clustering techniques such as swarm but I see huge value in Docker!

AngularJS: Architected and delivered a cutting edge data visualization system based on Angular 1.x, typescript,sccs,gulp. 
Introduced AngularJS in to multiple smaller projects.

Typescript: This is a fantastic language, and now especially with all the bells and whistles in v2.1 (not least async/await for es5 targets). If you are writing any Javascript you need to learn this no one will ever convince me that a dynamically typed language is better than a statically typed language for starters, but with all the new Standards based features now baked in, it’s certainly taking the industry by storm, I can’t see how Babel will continue to fight for its place in the world alongside it.

Ionic2: I wrote another mobile app, I’ve done this in many languages to date, I started out with iPhones and xamarin c#, moved to objectiveC and java, and finally settled back on the Typescript/Angular2 based Ionic2 framework. It’s a pleasure to deal with, and with my other investments in the underlying stack it has become a natural fit.

Java8: Finally spend some time getting up to speed on the new jdk and it’s offerings. While not strictly Java8, I’m including Sprint Bootstrap, Wildfly10 Application server, CDI, JaxRS etc in this section.

Camel: Gained a basic understanding and working knowledge of the Camel EAI framework.

ActiveMQ: I debated about putting this one on here, all queues fulfil the same core requirements to pass messages right? But I did approach ActiveMQ from three different sides camel/c#/java, so that was interesting.

.NET 2017 I’m now informed about what’s coming down the line. Some interesting thing like C#7 (which I will admit I had to read twice before I saw the value in the language changes), better support for the web stacks (although I’d admit with a tear in my eye that I’ve moved to Jetbrains software and am unlikely to come back to VStudio unless it’s an ASP based backend).

Client Products: Not only the development stacks have been changing, products in use by my clients have been moving at a rapid pace also and given they pay the bills I dedicate a fair amount of time to understanding them in depth.

Resource Consumption:
DNR -Listened to nearly every episode of DotNetRocks.
Hanselminutes - Funnily enough I found DotNetRocks as I used to subscribe the Hanselminutes; I say used to, as I’ve finally given up on Hanselminutes it appears to have moved in a different direction in the last year or so, don’t get me wrong, Scott is a great guy one of the best technical speakers in MS if you ask me, I even follow the weekly ASP.net stand-ups which he’s in, it only the podcast that I gave up on in December.

Other recommended podcasts:
Angular In Action
Javascript Jabber
RunAs Radio
Azure Podcast

2017

As you can image it takes a lot of time to get proficient in any of the above stacks I’ve mentioned . I’ve been trying to stay on top of them all and I’ve now reached the point or realization that need to let some go (think of Kate Winslet prying Leonardo DiCaprio’s icy fingers off that board she was on, it’ll be oh so sad). I’m going to narrow down the field, I’ll still keep in touch with them and if I encounter anything I don’t understand then I’ll make it my business to understand it, I simply won’t actively go pursuing them all. I’ve been burnt before with that approach, I learn Sliverlight after all, it wasn’t all bad as I wrote a windows phone app and many WPF apps around the same time, so the experience transferred nicely, it’s just that I’m not writing much WPF these days so I’ll put effort back in that direction only if and when needed.

Q: So the question remains: Where am I going to put my extended effort this year? 
A: Azure first approach. Azure will be the primary topic of my blogs whether the implementation is in C#/JavaScript/Typescript/Java I don’t really care. If the backend is .NET or Java, again I don’t really care but I do intend on blogging on practical usage cases for Azure services, I may even create a video or two!

Happy new year!

Not accepted here

and it's not because I'm Irish ;-)

ODataController - 406

So I’m working on a project that returns some entities from an OData endpoint, I wanted to remove the unnecessary parts of my entities so I created some DTO’s (which in itself is a good idea btw). The screenshot below shows the changes, I’m now returning a TimeSheetDto list (previously just a TimeSheet list) and I’m just projecting (using automapper) the Timesheets from my database into these DTO’s .

image

 

However when I test said entities I see nothing come back in my browser, nada,zilch,zippo… so what is happening?
Well I fired up fiddler and saw that I was getting a HTTP status code of 406, but why?
It worked before and I changed nothing…. oh wait!….

Configure OData entities

You know that message that gets scaffolded with new OData end points!? We’ll it explicitly tells us what entities are to be considered, so with a quick addition of my Dto Suffix in my OData config I’m back in action.

image

WebAPI OWIN and HppClient Authorize

Hi all, I know I promised my next post was going to be more Azure but I encountered a little task that took me a while to get working, the scenario was that I wanted to make a call to my WebAPI (MVC5) service using a C# HttpClient, the problem was that the resource I wished to access had the AuthorizeAttribute set

image

 

Now there’s a few ways to skin a cat but in the presence of the default Token Authorization one needs to first get a token and then use this token in subsequent requests. There is some good documentation using fiddler here: http://www.asp.net/web-api/overview/security/individual-accounts-in-web-api, however, there was not a lot of information on how to do this with HttpClient against Katana/Owin/MVC5, rather this information was not available in one specific place.

How

image

The first request gets the token and then this token is used as the Bearer for further requests.

Writing to an Azure Queue

If you've seen my previous post then this post is quite similar, this time however I write to an Azure Queue and not to a blob.

Code

First of all you need an Azure storage account as before, but once this is setup, consider the following code…

image

 

What I’m doing in the code above is

  1. Connecting to my storage account
  2. Creating the queue if it doesn't exist (remember you’ll get a bad request if you don’t name the queue correctly!).
  3. Then I create a simple message, I’m using an POCO object from another project and serializing it to JSON.

Did it work?

Lets use VS2013 U3 to check!

image

Open your server explorer and select the queue under the storage account you’ve chosen in your connection string, double click

image

Above you see the message added to the queue, you can see how many times it was de-queued  and when it’s set to expire, If we use a competing consumer pattern that count may be more than 1!

Next

I’m a little thorn re my next post, I’ve been writing a post on c# expression trees which is nearing completion, however I think to keep in line with the current trend I’ll post how this queue can be read and feed to an Azure Service Bus topic (pub/sub)… stay tuned ;-)

Uploading a Blob to Azure container

image

 

In the picture above you see a storage account in Azure, in the storage account we have an ecbfx (European Central Bank FX Rates) container. Now let’s see how to upload some data to this container using a C# console application.

NUGET

Given we are going to work with C# the best option is to use the .NET library, this can be retrieved from NuGet

image

 

Code

image

The code above connects to the pre-created container, notice that my container has built in Geo Redundancy (primary storage in Dublin, secondary in the Amsterdam) so after running there will be 6 copies of this blob, 3 in Dublin and 3 in Amsterdam, this is the storage package I’ve chosen.

image

 

View Blob

The easiest way to view the newly uploaded blob is to use the Windows Azure server explorer in Visual Studio, it’s the easiest way of getting the connection string to the storage account also.

image

In the next post I’m going to show you how react to someone uploading a Blob with an automatic trigger.

Angular.js .NET File Upload

In this post I’m going to show you how to upload a file using Angular.js on the client side and Asp WebApi on the back end.

Lets get started

Create you project in visual studio, and add your angular.js app controllers etc.

Interestingly enough I’ve already shown you how to do the server side over 2 years ago!
Crikey 2 years and I’m still writing about the same old stuff….. well not really, last time it was knockout, sliverlight and the likes, now it’s Angular.js’ turn.

Angular file upload, Nuget

In order to facilitate the process, we’re going to use a nuget package I like, see screenshot.

image

The beauty of this package is that its got shims for non html5 browsers (apparently there are a few hanging around still :-( )

To use this package you’ll need to include 2 scripts, file-upload-shim before angular.js and file-upload after.

Script Includes

image


Markup

Next add the input tag and add the ng-file-select directive

image

 

Javascript

Module

Add the upload module

Factory

image

Here i added the $upload factory to my controller

Controller function

image

Here I enumerate the files (should i wish to have multi select) then I upload each one by posting to my Web Api .net controller, I pass a little more information also as to the diff side, but that’s pretty much it.

.NET

Now even though I did show you the .net code before I’m going to show it again now, because as I mentioned I’m passing a little information as to the side the file I’m uploading represents.

image

image

Be careful of deferred execution

Take the following code, if you have ReSharper installed you’ll be warned that there is possible multiple enumeration of your IEnumerable, this means that  Select will be repeated twice for everything in the array.image

ToArray()

One solution is to Enumerate one and immediately after the select by calling .ToArrayimage

 

 

Guava / Java

It’s not just C# that you need to be careful with, take the google java library Guava

image We don’t get the same nice warning in IntelliJ* yet the solution in this case is much the same.

image

 

*IntelliJ

If you’re from a MS background like me and doing any Java, then do yourself a favour and use IntelliJ, it’s much easier to use.