Fastest way to read very large text file in C#

Tag: c#,wpf,large-files

I have a very basic question. I have several text files with data which are several GB's in size each. I have a C# WPF application which I'm using to process similar data files but nowhere close to that size (probably around 200-300mb right now). How can I efficiently read this data and then write it somewhere else after processing without everything freezing and crashing? Essentially whats the best way to read from a very large file? For my low scale application right now, I use System.IO.File.ReadAllLines to read and a streamwriter to write. I'm sure those 2 methods are not the best idea for such large files. I don't have much experience with C#, any help will be appreciated!

Best How To :

If you can do this line by line then the answer is simple

read a line
process the line
write the line

If you want it to go a bit faster then put those in three BlockingCollections with a specified upper bound of like 10 so a slower step is never waiting on a faster step. And if you can output to a different physical disc (if output is to disc).

OP changed the rules even after being asked if process line by line (twice)

read line(s) to generate unit of work (open to close tags)
process unit of work
write unit of work

Update list of items in c#

c#,linq,list,updates

I would do something like this: (for ordinairy lists) // the current list var currentList = new List<Employee>(); currentList.Add(new Employee { Id = 154, Name = "George", Salary = 10000 }); currentList.Add(new Employee { Id = 233, Name = "Alice", Salary = 10000 }); // new list var newList =...

How to return result while applying Command query separation (CQS)

c#,design-patterns,cqrs,command-query-separation

In such scenario I usually go with generating new entity Ids on the client. Like this: public class ProductController: Controller{ private IProductCommandService commandService; private IProductQueryService queryService; private IIdGenerationService idGenerator; [HttpPost] public ActionResult Create(Product product){ var newProductId = idGenerator.NewId(); product.Id = newProductId; commandService.AddProduct(product); //TODO: add url parameter or TempData key to...

Export data from table in Pervasive

c#,pervasive

No, there is no way to export data programmatically through the "Export Data" wizard in the Pervasive Control Center. You would need to either write or have written an export module.

deployment of a site asp.net and iis

c#,asp.net,iis

There are several domain providers like: godaddy, name etc you can use to buy a domain name. These providers also provide you steps to map the domain name to your website. Check out this link for example. This link explains domain name configuration in details.

Multiple Threads searching on same folder at same time

c#,multithreading,file-search

Instead of using ordinary foreach statement in doing your search, you should use parallel linq. Parallel linq combines the simplicity and readability of LINQ syntax with the power of parallel programming. Just like code that targets the Task Parallel Library. This will shield you from low level thread manipulation and...

Regex to remove `.` from a sub-string enclosed in square brackets

c#,.net,regex,string,replace

To remove all the dots present inside the square brackets. Regex.Replace(str, @"\.(?=[^\[\]]*\])", ""); DEMO To remove dot or ?. Regex.Replace(str, @"[.?](?=[^\[\]]*\])", ""); ...

Difference between application and module pipelines in Nancy?

c#,asp.net,nancy

The module- and application pipelines are explained in detail in the wiki. It's basically hooks which are executed before and after route execution on a global (application pipelines) and per-module basis. Here's an example: If a route is resolved to a module called FooModule, the pipelines will be invoked as...

Error when building an XDocument

c#,xml,linq,xpath,linq-to-xml

You can ignore pretty much all your code, the issue is just this: XDocument people = new XDocument("Persons"); You can't create an XDocument containing a string, you need to add an element: XDocument people = new XDocument( new XElement("Persons", original.XPathSelectElements(xpathFilterDups))); ...

while Inherit style in WPF it affect parent style

c#,xaml,styles,wpf-controls

If you declare a Style without an x:Key, it will override the default style for that control. <Style TargetType="local:CustomControl"> So the code above will effect all CustomControl elements throughout the entire application (or within the scope). If you do not want to override the base style, you can give your...

Visual Studio Assembly force-installs Target Framework

c#,.net,visual-studio-2013,.net-framework-version

The targeted .NET version is the only version that the app will depend upon by default. Visual Studio will not automatically add higher and backwards compatible releases. Do this manually by adding other .NET versions to a configuration file: On the Visual Studio menu bar: Choose Project; Add New Item;...

Foreign key in C#

c#,sql,sql-server,database

You want create relationship in two table Refer this link http://www.c-sharpcorner.com/Blogs/5608/create-a-relationship-between-two-dataset-tables.aspx...

C# - Can't connect to remote MySQL server

c#,mysql

When connecting to a MySQL-Database I always used the MySQL Connector you can get here: https://dev.mysql.com/downloads/connector/net/6.9.html You have to import the MySQL namespaces to your project and then you can use the MySQLConnection instead of the SQLConnection that is, as far as I know, only for MSSQL servers. http://www.codeproject.com/Tips/423233/How-to-Connect-to-MySQL-Using-Csharp...

Marshal struct in struct from c# to c++

c#,c++,marshalling

Change this: [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 36)] private string iu; to this: [MarshalAs(UnmanagedType.LPStr)] private string iu; Note that this code is good only to pass a string in the C#->C++ direction. For the opposite direction (C++->C#) it is more complex, because C# can't easily deallocate C++ allocated memory. Other important thing:...

Aligning StackPanel to top-center in Canvas

c#,wpf,xaml,canvas

If you don't want any input or hit testing on a certain element you should set the IsHitTestVisible property to false: <Grid> <Canvas Name="Canvas" Background="#EFECCA"> <DockPanel VerticalAlignment="Stretch" HorizontalAlignment="Stretch" Width="{Binding ActualWidth, ElementName=Canvas}" Height="{Binding ActualHeight, ElementName=Canvas}" MouseLeftButtonDown="DockPanel_MouseLeftButtonDown" TouchDown="DockPanel_TouchDown" Panel.ZIndex="2" Background="Transparent"> </DockPanel> <Button Width="50" Height="50"...

How to send Ctrl+S through SendKeys.Send() method to save a file(save as dialog)

c#,.net,windows,sendkeys

I believe you need to use: SendKeys.SendWait("^(s)"); Instead of: SendKeys.SendWait("^%s?"); Have a look at https://msdn.microsoft.com/en-us/library/system.windows.forms.sendkeys.send(v=vs.110).aspx for more information....

Changing color of positionmarker/caret

wpf,wpfstyle

To modify the color of the caret (which assume is what you call "positionmarker") you have to set the CaretBrush property of the TextBox.

Is it possible to concactenate a DataBound value with a constant string in XAML DataBinding?

c#,xaml,windows-phone

You can use a StringFormat in your binding, like so: <TextBox Text="{Binding ItemName, StringFormat={}Item: {0}}"/> That being said, it may cause some unexpected behavior when editing. For example, if the user edits only the item name (excluding the 'Item:' text), then when the TextBox loses focus, the string format will...

Catch concurrency exception in EF6 to change message to be more user friendly

c#,asp.net,.net,entity-framework,entity-framework-6

You are executing an asynchronous method. This means that any exceptions will be thrown when you call await on the returned task or when you try to retrieve the results using await myTask; You never do so, which means that the exception is thrown and caught higher up your call...

SQL Server / C# : Filter for System.Date - results only entries at 00:00:00

c#,asp.net,sql-server,date,gridview-sorting

What happens if you change all of the filters to use 'LIKE': if (DropDownList1.SelectedValue.ToString().Equals("Start")) { FilterExpression = string.Format("Start LIKE '{0}%'", TextBox1.Text); } Then, you're not matching against an exact date (at midnight), but matching any date-times which start with that date. Update Or perhaps you could try this... if (DropDownList1.SelectedValue.ToString().Equals("Start"))...

How to declare var datatype in public scope in c#?

c#,linq

Declare it as a known type (not an anonymous type), like this for example: Dictionary<int, string> results = new Dictionary<int, string>(); Then you could store the results in the Dictionary: results = behzad.GAPERTitles.ToDictionary(x => x.id, x => x.gaptitle); And reference it later: private void button1_Click(object sender, EventArgs e) { //...

MahApps - How to disable automatic uppercase of default button

wpf,mahapps.metro

You can override the default value by setting the property for all buttons in Window.Resources <controls:MetroWindow ... xmlns:controls="clr-namespace:MahApps.Metro.Controls;assembly=MahApps.Metro" xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation" xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"> <Window.Resources> <ResourceDictionary> <Style TargetType="{x:Type Button}" BasedOn="{StaticResource {x:Type Button}}"> <Setter Property="controls:ButtonHelper.PreserveTextCase"...

Collect strings after a foreach loop

c#,xml,foreach

Yep, you need to do the adding within the loop. I'd use a List<string> as it supports LINQ: XmlNodeList skillNameNodeList=SkillXML.GetElementsByTagName("name"); List<string> skills = new List<string>(); foreach (XmlNode skillNameNode in skillNameNodeList) { skills.Add(skillNameNode.Attributes["value"].Value); } ...

Validate a field only if it is populated

c#,wpf,idataerrorinfo

You can implement your OptionalPhoneAttribute based on the original PhoneAttribute: public sealed class OptionalPhoneAttribute : ValidationAttribute { public override bool IsValid(object value) { var phone = new PhoneAttribute(); //return true when the value is null or empty //return original IsValid value only when value is not null or empty return...

Memory consumption when chaining string methods

c#,string,immutability,method-chaining

Is it true that when you chain string functions, every function instantiates a new string? In general, yes. Every function that returns a modified string does so by creating a new string object that contains the full new string which is stored separately from the original string. There are...

How do I provide a collection of elements to a custom attached property?

c#,wpf,binding

I managed to get it working using an IMultiValueConverter like this: public class BorderCollectionConverter : IMultiValueConverter { public object Convert(object[] values, Type targetType, object parameter, System.Globalization.CultureInfo culture) { var borderCollection = new BorderCollection(); borderCollection.AddRange(values.OfType<Border>()); return borderCollection; } public object[] ConvertBack(object value, Type[] targetTypes, object parameter, System.Globalization.CultureInfo culture) { throw new...

Unconstrained type parameters casting

c#,.net,types,casting

The compiler sees the T2 and T identifiers and helpfully informs you that those types seem unrelated. That's absolutely correct, as they have no relation: there are no generic constraints that would assert any relations between them (I'm not saying that would be useful here though :) ). Whether this...

System.net.http.formatting causing issues with Newtonsoft.json

c#,asp.net,asp.net-mvc,json.net

Does the assemblyBinding tag have proper xmlns schema? Check if the issue you are encountering is same as Assembly binding redirect does not work

Index was out of range. Must be non-negative or less than size of collection [duplicate]

It looks like you have a typo in your loop condition: for (int index = filePaths.Count(); filePaths.Count() > 9; index--) It should be for (int index = filePaths.Count() - 1; index > 9; index--) Also note that for the first iteration of loop you're trying to access filePaths[filePaths.Count()] which is...

Regex that allow void fractional part of number

c#,regex

Just get the dot outside of the captruing group and then make it as optional. @"[+-]?\d+\.?\d*" Use anchors if necessary. @"^[+-]?\d+\.?\d*$" ...

C# Code design / Seperate classes for each TabControl

c#,oop,architecture,software-design,code-design

Place a UserControl on each tab.

C# XML: System.InvalidOperationException

c#,xml

Is "User Info" and "Course Data" is a different entity. If it is so, I think you may encapsulate them in one entity. XmlTextWriter writer = new XmlTextWriter(path, System.Text.Encoding.UTF8); writer.WriteStartDocument(true); writer.Formatting = Formatting.Indented; writer.Indentation = 4; writer.WriteStartElement("My Entity"); /* It is a biggest one*/ writer.WriteStartElement("User Info"); writer.WriteStartElement("Name"); writer.WriteString(userName); writer.WriteEndElement(); writer.WriteStartElement("Tutor...

Why is the task is not cancelled when I call CancellationTokenSource's Cancel method in async method?

c#,asynchronous,task,cancellationtokensource,cancellation-token

Cancellation in .Net is cooperative. That means that the one holding the CancellationTokenSource signals cancellation and the one holding the CancellationToken needs to check whether cancellation was signaled (either by polling the CancellationToken or by registering a delegate to run when it is signaled). In your Task.Run you use the...

C# MySQL Parameters.AddWithValue

c#,mysql

You try to add all your 52 parameter and their values with one AddWithValue method. You can't do that. First of all, you need to define all your parameters in your command with your column names like; command.CommandText = "INSERT INTO tb_mitarbeiter (Vorname, id, projectnummber....) VALUES (?name, ?id, ?projektnummer....)"; Then...

How to Customize Visual Studio Setup

c#,visual-studio,setup-project

You can use a Microsoft Setup project or WIX (easily integrate with Visual Studio). Both are free. •You can do almost all of your customization in setup project by adding custom actions. •WIX (window installer xml) is the better option. You can do a complete customization from wix but it...

Load XML to list using LINQ [duplicate]

c#,xml,linq

Make a base class which will have id,x,y,z, and have Vendors,Bankers and Hospitals extend it. Then you can have a collection of the base class, and add to it the classes that inherit from it....

Convert Date Time to IST

You need to use "India Standard Time" instead of "Indian". Please refer to this link for a list of the time zone descriptions.

how can I add a column to IQueryable object and modify its values

c#,.net,linq,grid,devexpress

Simple example for using a non-anonymous class. public class MyLovelyClass { public Int32 Number { get; set; } public bool Selection { get; set; } } var packs = from r in new XPQuery<Roll>(session) select new MyLovelyClass() { Number = r.number }; gcPack.DataSource = packs; ...

DialogBox with value verifications

c#,.net,winforms

The problem is you're trying to enable or disable the button when checking individual textboxes and they're conflicting with each other, instead the logic needs to be at a higher level. Change your textbox validation function to return a bool, and use that in ValidateAll to determine whether or not...

check if file is image

c#,asp.net,asp.net-mvc

You can't do this: string.Contains(string array) Instead you have to rewrite that line of code to this: if (file == null || formats.Any(f => file.Contains(f))) And this can be shortened down to: if (file == null || formats.Any(file.Contains)) ...

C# PCL HMACSHAX with BouncyCastle-PCL

c#,bouncycastle,portable-class-library

Try like this for HmacSha256 public class HmacSha256 { private readonly HMac _hmac; public HmacSha256(byte[] key) { _hmac = new HMac(new Sha256Digest()); _hmac.Init(new KeyParameter(key)); } public byte[] ComputeHash(byte[] value) { if (value == null) throw new ArgumentNullException("value"); byte[] resBuf = new byte[_hmac.GetMacSize()]; _hmac.BlockUpdate(value, 0, value.Length); _hmac.DoFinal(resBuf, 0); return resBuf; }...

Convert contents of an XmlNodeList to a new XmlDocument without looping

c#,xml,xpath,xmldocument,xmlnodelist

If you're happy to convert it into LINQ to XML, it's really simple: XDocument original = ...; // However you load the original document // Separated out for clarity - could be inlined, of course string xpath = "//Person[not(PersonID = following::Person/PersonID)]" XDocument people = new XDocument( new XElement("Persons", original.XPathSelectElements(xpath) )...

How can I determine if an object of anonymous type is empty?

c#,.net

Anonymous types do not provide operator overloads for ==, although it wouldn't matter in this case since one of the arguments is typed object. However the C# compiler does provide Equals, GetHashCode, and ToString implementations. Use the static object.Equals, method which will do the appropriate null checks and then call...

Access manager information from Active Directory

c#,asp.net,active-directory

try this: var loginName = @"loginNameOfInterestedUser"; var ldap = new DirectoryEntry("LDAP://domain.something.com"); var search = new DirectorySearcher(ldap) { Filter = "(&(objectCategory=person)(objectClass=user)(sAMAccountName=" + loginName + "))" }; var result = search.FindOne(); if (result == null) return; var fullQuery = result.Path; var user = new DirectoryEntry(fullQuery); DirectoryEntry manager; if (user.Properties.PropertyNames.OfType<string>().Contains("manager")) { var managerPath...

MvcSiteMapProvider - Enhanced bootstrap dropdown menu

c#,twitter-bootstrap,asp.net-mvc-5,mvcsitemapprovider,mvcsitemap

node.Descendants should be node.Children Learn the difference on Descendants and Children here, CSS Child vs Descendant selectors (Never mind the post beeing about CSS, this is a generic pattern)...

Get object by attribute value [duplicate]

c#,reflection,custom-attributes,spring.net

If you have obtained the Assembly, you can just iterate over the types and check for your conditions: var matchingTypes = from t in asm.GetTypes() where !t.IsInterface && !t.IsAbstract where typeof(ICustomInterface).IsAssignableFrom(t) let foo = t.GetCustomAttribute<FooAttribute>() where foo != null && foo.Bar == Y select t; I am assuming you want...

SettingsProvider class - should it be in DAL or BLL project?

c#,data-access-layer,bll

I don't agree that there is a right layer for you to put that class since you reading values from the config file based on keys provided and it can be needed by one or all of the layers. In the case of all layers using this class, you can...

Call Sync method call from Async Callback?

c#,asynchronous,synchronous

Callbacks are independent as they're invoked on the thread-pools IO completion workers. If you're interested, you can see that in the source code. This particular method is for the Socket class (which TcpClient and UdpClient use internally), where overlapped IO is used to invoke the callback (see the comment on...